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Introduction 


This project is broken into two focus areas: telesurgery and surgical rehearsal. In each we are 
exploring various applications and extensions of the existing robotic surgical systems. Under 
telesurgery we are exploring the ability to perform telesurgery using a robot both across a state- 
wide and a nation-wide area based on the currently available technology. Under surgical 
rehearsal we are exploring designs for simulator systems which can be used to improve training 
and education of surgeons pursuing expertise in the use of robotic surgical systems. The focus is 
on unique forms of robotics which have not previously been addressed by simulation 
technologies. 


The original term of this project was September 2014 to August 2016. However, on April 6, 
2016, Florida Hospital requested a no-cost extension of 6 months to the term of the agreement, 
for a new end date of February 28, 2017. On May 13, 2016, the government granted this 
extension. Therefore, this annual report covers the period from September 2015 to August 2016, 
but does not align with the end of the project. A separate final report will be filed by March 30, 
2017 to terminate the entire agreement. 


Reporting Period: September 1, 2015 to August 31, 2016. 
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Statement of Work 


Original Term of Project 


Telesurgery: Metropolitan Latency. Perform robotic surgical experiments between multiple 
campuses within a metropolitan area, between campuses across a state area, and across 
nationwide campuses. 
Period 1 Milestone: Telesurgery state-wide latency data and report. Award + 360 days. 
Period 2 Milestone: Telesurgery nationwide latency data and report. Award + 700 days. 


Surgical Rehearsal. Develop virtual reality environment for training operating room staff in 
robotic surgery. Develop design for simulators in hard-tissue robotic surgery (spinal and 
orthopedic). 

Period 1 Milestone: Spinal simulator design document. Award + 300 days. 

Period 2 Milestone: OR team training virtual world environment. Award + 360 days. 

Period 2 Milestone: Orthopedic surgery rehearsal validation report. Award + 720 days. 


Modifications from May 2015 Extension 


Surgical Rehearsal. Develop design for simulators in hard-tissue robotic surgery (spinal and 
orthopedic). 

Milestone 1: Spinal simulator design document. February 28, 2107. 

Milestone 2: Orthopedic simulator design document. February 28, 2107. 
Evaluating Simulator Metrics. Compare the metrics assigned by expert surgeons to those 
assigned by the simulator software. 

Milestone 1: Simulator Metric Evaluation Document. February 28, 2107. 
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Project Management 


Progress Summary. 

* OR Team Training Virtual World has been completed and the application has been 
posted on the open internet for world-wide access. See www.TrainRobotic.com . 

¢ Telesurgery experiments with all sites are complete. 

* Robotic Spinal Surgery Simulator system and user analys 
the final design document to be delivered in February 2017. 

© Robotic Orthopedic Surgery Simulator system and user analysis are complete, We are 
writing the final design document to be delivered in February 2017. 

¢ Evaluation of simulator metrics is processing subjects through the experiment. 


are complete. We are writing 


Schedule. 
This schedule shows our expected progress and completion of the experiments remaining on this 
grant. 


Category _| Project yags | ysqi | ysq2 | ysq3_| ysqa yeqi | y6qz 
Simulator Evaluations 
Simulator Perform 
Eval Sin Metics — 
Robotic Simulator Design 

OR Virtual World 
Spinal Robotics 
Orthopedic Robotics 


Telesurgery 


Comms Latency 
indicates current time at report 


Budget. 
Following the extension of the term of the project, financial spending on the project is aligned 
with the expected task and schedule completion for the project. 
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Scientific Progress 


Simulator Performance. 


Human subject data collection is finished. Expert surgeons were the most challenging population 
to collect. We collected data from Celebration Health, Columbia University Medical Center, and 
at the annual meeting of the Society for Laparoscopic Surgeons. 


We are beginning full analysis of the data from all populations included in the study. The results 
of this work were presented at the 2015 /ITSEC conference and the 2016 IEEE Serious Games 
in Healthcare Conference; served as the basis for one doctoral dissertation; and have been 
published in the Journal of Surgical Endoscopy. 


A basic plot of the performance of four different populations while performing the Suture 
Sponge exercise is provide below (lower scores indicate higher skill levels in “economy of 
motion” and “time to complete”). This shows a very distinct performance difference between the 
expert surgeons and all other populations. It also appears that there is little difference between 
the population of lay people, medical students, and video gamers. The conclusion of the study is 
that subjects who are expert video game players do not exhibit skills in surgical simulation 
which are equivalent to practicing robotic surgeons. Rather, their performance is similar to all 
other subject categories who have little/no robotic surgery experience. 
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Evaluation of Simulator Metrics 


We are processing subjects through this experiment. We currently have 23 subjects with a goal 
of 70 for completion. A preliminary analysis of the data from the current subjects has been 
performed and will be presented at the annual AAGL Congress in November 2016. 
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da Vinci OR Virtual World 


This virtual world has been completed and made publicly available for access on the open 
internet. Interested parties may access and use the application at no cost. The hosting fees for the 
application will be paid by Florida Hospital to make it accessible to the public. 


Users may choose to create a unique account so their personal scores can be tracked and reported 
to their institution. Or they may choose to use it through a “guest” account in which performance 
is not scored. 


The application can be accessed at: 


‘A paper on this project was presented at the 2015 I/ITSEC conference. The finished game has 
been entered into the 2016 I/ITSEC Serious Games Showcase and Challenge. 
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Spinal Robotic Simulator 


We have completed our analysis of the Mazor Spinal Robotic system as well as our analysis of 


the potential trainees. We are preparing the simulator design document for delivery in February 
2017. 
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Mazor Simulator Assessment Strategy 


skill Objective Domain Item/Criteria Media & Tool 
Selection 

Properly upload/open | Demonstrate the | Knowledge | Choose the option to upload patient specific CTs from Virtual 
patient's imaging procedure needed their own collection or chooses a certain CT pertaining to 

to import unique pathology they wish to manipulate. Use planning tab and 

CT scans or open CT upload icon from the system. CT must follow Mazor’s 

scan from protocol (i.., low-dose, Imm contiguous axial slices). 

exercise/patient Choose to open pre-existing cases from the planning tab in 

data base. the system. 
Select the Region of | Determine and Knowledge | Use the steps below to select ROI: Virtual 
Interest (RO!) select the correct 1. Drawalline on the correct vertebral bodies. 

vertebral bodies for 2. Include a body above and below ROI 

given pathology. 3. Mark path of vertebral foramen (in both AP and 

lateral view) 

Appropriately Determine the Knowledge | Separate the vertebral column into vertebral units. Virtual 
perform ROI vertebral units of Segmentation should not cut through end plates. 
segmentation. the ROI 
Correctly label Accurately label__| Knowledge | Select and label one vertebra in the ROI to populate the _ | Virtual 
vertebrae vertebrae remaining vertebra in ROI. 
Choose the proper | Accurately choose | Knowledge | Choose place screw option. Incorrect hardware (e.g., Virtual 
hardware for the proper screws) will not appropriately fit the pathology when 
preoperative plan hardware needed shown 

for case specific 

images. 
Correctly place Determine and Application | Use the steps below: Virtual 
hardware during place the hardware 1. Place the screws. 
preoperative planning | in the preoperative 2. Change implant parameters per case. 

stage. 3. Use the software to play through 1mm slices of the 

image to accurately view placement. 
4, Leamer adjusts if screw placement looks incorrect. 

Identify correct Determine the Knowledge | Chooses a platform that is appropriate to successfully Physical Model 
platform platform indicated complete the operative stage. 

by patient's 

pathology. 
Properly place the | Complete the Application | Use the steps below to register the intraoperative image: | Virtual/Physical 
registration 3D required 1. Calibrate C-Arm using the correct marker. Model 
marker registration process 2. Properly attach the 3D marker. 

to match 

preoperative CT 

with intraoperative 

fluoro image. 
Accurately determine | Determine if Analysis | Learner must examine image using the following: Virtual 
an appropriate fluoro | intraoperative 1. The ROIs clearly shown in the image. 
image using RBC image is acceptable 2. Allof the beads from the 3D marker are 

for image captured 

registration. 3. Image shows clear vertebral anatomy 
Properly mount Demonstrate Application | Depending on the platform chosen, learners will Physical Model 
chosen platform procedural 1. Choose the correct corresponding tools and 
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knowledge for 
‘mounting the 
correct platform. 


hardware. 

2. Determine which part of the bony anatomy to 
mountto. 

3. Mount parallel to the spine. 

4. Mount platform using the correct order of, 
operations. 

5. Tighten from a neutral position. 


Use guidance system | Using registered | Application 1. The learner should ensure the preoperative plan | Virtual/Physical 
image and is correct. Model 
preoperative plan, 2. Gather tools needed (given by software, 
execute the determined by preoperative plan) 
guidance system. 3. Place robot using provided navigation, 

4. Execute the robot using the software. 
5. Wait for the robot to stop moving (or when the 
blinking stops) 

Place hardware (i.e., | Determine correct | Application | Depending on case/exercise: Physical 

cannulas) hardware, 1. Use simulated physical or virtual hardware to | Model/Virtual 
demonstrate indicate placement of cannulas. 
hardware 2. Interpret tactile feedback to adjust or maintain 
placement. positioning, 

Properly use surgical | Demonstrate the | Application 1. Ensure drill doesn't skive. Physical 

tools (ie., drill, k- procedural 1.1 Ifso, remove tissue using simulated Model/Virtual 


wires, and screws.) 


knowledge and 
skills to place the 
instruments for the 
procedure. 


hammer 
2. Interpret tactile feedback to adjust or maintain 
positioning, 

3. Remove drill. 

4, Place k-wires. 

5. Insert correct hardware. 

6. Continue until all trajectories are reached. 
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Orthopedic Robotic Simulator 


We have completed our analysis of the Mako Orthopedic Robotic system as well as our analysis 
of the potential trainees. We are preparing the simulator design document for delivery in 
February 2017. 


‘enables hene-soaring 
teutacr 


yinasive 7] 
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Telesurgery. 


Orlando-to-Seattle. Working with the surgical robotics team at University of Washington, led by 
Dr. Blake Hannaford, we were able to use existing experiments performed by UW to estimate 
the probable latency between the two sites. Based on this previously performed and published 
work, we estimate the latency to fall between 30ms and 75ms. 


When compared with the data collected to Denver and Fort Worth, the Seattle estimate appears 
to be more aggressive or optimistic. We believe this is driven by two factors. UW’s experiments 
were performed using the UW Raven robotic device and represent video data of a lower 
resolution. This could easily account for the differences in the results. 


Telesurgery Latency and Safety 


Seat 
‘estimated 
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Key Research Accomplishments 


* Telesurgery: Communications Latency. Major hospital systems have sufficient 
telecommunication bandwidth to perform robotic telesurgery right now. 

© Surgical Rehearsal. Simulation-based training for different forms of robotic procedures 
appears to be feasible beyond the simulators of the da Vinci robot which have previously 
been created. These could be applied to systems like the Mazor Renaissance spinal 
robotic system and the Mako Rio orthopedic robotic system. 

Multiple publications and presentations have been generated from this research work. 


Annual Report, September 2016. 
Project W81XWH-11-2-0158 Medical Robotic and Telesurgical Simulation and Education Research 


Reportable Outcomes 


(since September 2015 only) 


Publications 
1. Julian, Tanaka, Mattingly, Perez, Truong, Simpson, Smith. “Comparative Analysis of Four 
Simulators of the da Vinci Surgical Robot”, Surgical Endoscopy, (Under review). 


2. Smith, Tanaka, Julian, Mattingly. (Dec 2016). “Blended Training for Surgeon Education.” 
2016 Interservice/Industry Training Education and Simulation (/ITSEC) Conference. 


3. Julian, Tanaka, Mattingly, Smith. (Dec 2016). “Surgical Simulator Showdown.” 2016 
Interservice/Industry Training Education and Simulation (I/ITSEC) Conference. 


4, Tanaka, Julian, Mattingly, Smith. (Dec 2016). “Validation for Simulators: It’s All About 
Perspective.” 2016 Interservice/Industry Training Education and Simulation (I/ITSEC) 
Conference. 


5. Smith R. “Response to Unlike History, Should a Simulator Not Repeat Itself?” Simulation in 
Healthcare, 11(3). 


6. Mouraviev et al. “Robotic training with porcine models induces less workload than virtual 
reality robotic simulators for urology resident trainees” Journal of the AUA and AUA Annual 
Congress, Dec 2015 


7. Smith, Tanaka, Mclllwain, Willson. (Dec 2015). “Developing Game-based Leadership 
Training for Robotic Surgeons.” 2015 Interservice/Industry Training Education and 
Simulation (I/ITSEC) Conference. 


8. Tanaka, Graddy, Smith, Perez. (Dec 2015). “Gamers Today, Surgeons Tomorrow?” 2015 
Interservice/Industry Training Education and Simulation (I/ITSEC) Conference. 


9, Tanaka, Graddy, Simpson, Perez, Truong, & Smith. (Accepted). “Robotic Surgery 
Simulation Validity and Usability Comparative Analysis”. Journal of Surgical Endoscopy. 


Presentations 

1. Mattingly, Tanaka, Julian, Smith. (Nov 2016). “Virtual Reality Robotic Simulation 
Performance Assessment: Simulator Metrics vs. GEARS.” Annual Meeting of the American 
Association of Gynecologic Laparoscopists. 


2. Mattingly, Tanaka, Julian, Skinner, Smith. (Nov 2016). “Simulator-based Multi-modal Task 
Decomposition of Robotic Surgical Technique for Vaginal Cuff Closure.” Annual Meeting of 
the American Association of Gynecologic Laparoscopists. 
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3. Tanaka, Smith. “Searching for Cognitive Ergonomics in Surgical Education”, International 
Meeting on Human Factors and Ergonomics, June, 2016. 


4. Smith R. “R&D in Robotic Surgery Devices”, Annual Meeting of the Industrial Research 
Institute, May 2016. 


5. Smith, “Robots in Surgery and Simulation in Training.” IEEE International Systems 
Conference, April 2016. 


6. Julian, Tanaka, Smith, “A Side-by-Side Comparison of Virtual Reality Robotic Surgical 
Simulators.” Florida Hospital Research Conference, April 2016. 


7. Tanaka, Smith, Hughes, “Video Game Experience and Basic Robotic Skills.” IEEE 
International Conference on Serious Games and Applications for Health, February 2016. 


8. Smith R. “The Validation of Surgical Simulators for RASD”. FDA Workshop on Robotically 
Assisted Surgical Devices, Washington DC, August 2015. 


9, Tanaka, Graddy, Perez, Simpson, Truong, Smith. (Nov 2015). “Video Game Impact on Basic 
Robotic Surgical Skills.” Annual Meeting of the American Association of Gynecologic 
Laparoscopists. 


10. Perez, Tanaka, Simpson, Truong, Smith, Satava. (Nov 2015). “From concept to surgical 
relevance: Engineering the training device for the Fundamentals of Robotic Surgery.” Annual 
Meeting of the American Association of Gynecologic Laparoscopists. 


Poster Presentations 

1. Julian, D., Tanaka, A., Mattingly, P., & Smith, R. (2016). “A Side-by-Side Comparison of 
Virtual Reality Robotic Surgical Simulators.” University of Central Florida Graduate 
Research Forum 2016. Orlando, Fl. 


2. Tanaka, Graddy, Perez, Simpson, Truong, Smith. (Jan 2016). “Video Game Impact on Basic 
Robotic Surgical Skills.” International Meeting on Simulation in Healthcare, San Diego, CA. 


Annual Report, September 2016. 
Project W81XWH-11-2-0158 Medical Robotic and Telesurgical Simulation and Education Research 


Conclusion 


Each of the research areas funded by this grant has made significant scientific contributions. The 
knowledge gained from this work is being shared through reports to the government and multiple 
presentations at both clinical and simulation conferences. We have also submitted multiple 
papers for journal publication. 


This cooperative agreement is scheduled to end on February 28, 2017. Based on our current 
work flow and state of funds the project is currently on schedule to complete all objectives by the 
end of the agreement. 
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Appendices 


Copies of manuscripts, abstracts, and presentations of work resulting from this grant are included 
as appendices to this report. 
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Developing Game-based Leadership Training for Robotic Surgeons 


Roger Smith & Alyssa Tanaka Steve Mellwain & Brad Willson 
Florida Hospital Nicholson Center ARA/Virtual Heroes 
Celebration, FL, Raleigh, NC 
Roger.Smith@flhosp.org Steve.Mcllwain@ virtualheroes.com 


‘ssa. Tanaka@ flhosp.org Bradley.Wi 


ABSTRACT 


Al surgeons must simultaneously perform as skilled practitioners and effective team leaders in the operating 
room. This is further complicated in robotic surgery because the surgeon is removed a short distance from the 
operating table and works from within a specialized cockpit. This separation creates a unique hurdle when a 
crisis arises that requires the surgeon to disengage from the immediate steps of the surgery to provide leadership 
and guidance with issues involving the team, the equipment, the room, or the patient. 


To develop and test these skills we initially created a series of scenario-based videos with quizzes to evaluate 
surgeon understanding of these leadership responsibilities. Using these as a guide, we developed a game-based 
virtual environment containing the same information as the videos but in a 3D interactive space which is 
accessible through a web browser. This environment presents accessible and engaging scenarios that include a 
scoring mechanism which can assess the time to react to events, the actions that occur before and after a 
decision, and the correctness of the decision made. The tool can also present alternative or repetitive scenarios 
when the student does not take the correct action. This paper describes the development process and the 
interactions with the surgeons and operating room teams which drove the design and content of the virtual 
environment. The paper also describes the longer term plans to validate the content and introduction of the game 
to multiple surgical training sites around the country. Though the virtual environment uses a more interactive 
method for presenting leadership and team decision making information, we are interested in whether it is more 
effective than traditional didactic lectures, textual instructions, videos, and live role playing 


ABOUT THE AUTHORS, 


Roger Smith is an expert in the development of simulation devices and training programs. He has spent 25 years 
creating leading-edge simulators for the Department of Defense and Intelligence agencies, as well as accredited 
methods for training with these devices. He is currently the Chief Technology Officer for the Florida Hospital 
Nicholson Center where he is responsible for establishing the technology strategy and leading research 
experiments. He has served as the CTO for the U.S. Army PEO for Simulation, Training and Instrumentation 
(PEO-STRD; VP and CTO for training systems at Titan Corp; and Vice President of Technology at BTG Inc. He 
holds a Ph.D. in Computer Science, a Doctorate in Management, and an M.S. in Statistics. He has published 3 
professional textbooks on simulation, 10 book chapters, and over 100 journal and conference papers. His most 
recent book is A CTO Thinks About Innovation. He has served on the editorial boards of the Transactions on 
Modeling and Computer Simulation and the Research Technology Management journals. 


Alyssa D.S. Tanaka is a Systems Engineer at Florida Hospital Nicholson Center. Her research work focuses on 
robotic surgery simulation and effective surgeon training. Her current projects include rapid prototyping of 
surgical education devices, the validation of a robotic surgical curriculum and evaluation of robotic simulation 
systems. She is a Modeling and Simulation PhD student at the University of Central Florida and previously 
eamed a M.S. in Modeling and Simulation, Graduate Simulation Certificate in Instructional Design, and a B.S. 
in Psychology and Cognitive Sciences from the University of Central Florida. She holds a diploma in robotic 
surgery from the Department of Surgery, University of Nancy, France. 


Steve Mellwain is a Senior Producer for the Virtual Heroes Division of Applied Research Associates, Inc. He 
has over 10 years of production experience in the 3D animation and interactive entertainment industries. Prior to 
joining Virtual Heroes, Steve worked at Walt Disney Feature Animation and Blizzard Entertainment. He 
specializes in production management, macro/micro scheduling, team building, and finance. He is passionate 
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about creating virtual worlds that educate, inform, and inspire. Steve holds a B.S. in Marketing and an M.B.A. 
from Azusa Pacific University 


Bradley Willson is the Game Design Lead for the Virtual Heroes Division of Applied Research Associates, Inc. 
A nine-year veteran of the game industry, Brad began his career at Rockstar San Diego as a Development 
Support Supervisor, where he worked on various commercial game titles including the Midnight Club series, 
Table Tennis, and Red Dead Revolver. He joined Virtual Heroes in 2006, with the goal of incorporating his 
commercial game experience into games that had a true altruistic focus. At Virtual Heroes, Brad works to create, 
drive, and deliver the overall creative vision for the numerous serious games in development. Brad creates 
compelling software designs from conflicting viewpoints, communicates the designs to the development team, 
and translates the designs into detailed software mechanics, gameplay progression, and interface flow. Brad 
holds a B.S. in Wildlife Science from Purdue University. 
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Developing Game-based Leadership Training for Robotic Surgeons 


Roger Smith & Alyssa Tanaka Steve Mellwain & Brad Willson 
Florida Hospital Nicholson Center ARA/Virtual Heroes 
Celebration, FL, Raleigh, NC 
Roger.Smith@flhosp.org Steve.Mcllwain@ virtualheroes.com 
Alyssa.Tanaka@ flhosp.org Bradley. Willson @ virtualheroes.com 


INTRODUCTION 


Surgery is a team sport requiring the coordinated activities of multiple healthcare professionals. This team 
assembles daily in different combinations for a few hours with the chief surgeon as the leader who is responsible 
for directing the surgical activity. Historically, the surgeon has been the most highly educated member of the 
team, the most socially respected, and the most dominant personality. This has created situations in which the 
surgeon manages the team as a dictator who does not listen to the experience-based concerns and educated input 
from the other members of the team. Organizations like the American College of Surgeons and the World Health 
Organization have responded to these issues by creating and propagating standard practices and training 
materials which promote cooperative participation by all members of the team and an open, inclusive attitude by 
the surgeon/leader of the team. The surgeon remains the primary person responsible for the outcome of the 
surgery, but is encouraged or required to solicit and apply the expertise of all members of the OR team. 


Robotic surgery with the da Vinci surgical robot, the dominant device in the field, introduces additional 
challenges for keeping a team working together. Changes in the physical location and orientation of team 
members create one new hurdle in team cooperation. Figure 1a illustrates the positions of typical members of a 
surgical team for open and laparoscopic procedures, Everyone is physically clustered around the patient, within 
arm’s reach and easy speaking distance of one another. Direct eye-to-eye contact and communication is easy and 
directives to the team are difficult to confuse. By contrast, Figure 1b illustrates the positions of members of a 
robotic team. Most members remain at the bedside, but the surgeon has been separated from the encircled group. 
In order to operate the robot, the surgeon must remove himself from the bedside and take a position within a 
custom console to control the machine. This console pulls their physical actions, visual attention, and mental 
focus into an environment that is separate and unique from the rest of the team. This situation can potentially 
undermine the previous work that has been done to integrate the actions and expertise of the team within more 
traditional forms of surgery. 


‘The manufacturer of the da Vinci robot has attempted to mitigate this separation by including a microphone and 
speakers in the head-space of the robotic console. So the words spoken by the surgeon are broadcast to the rest 
of the team from speakers attached to the bedside components of the robot. Similarly, a microphone on the 
bedside equipment captures the discussions of the surgical team and carries it to speakers in the surgeon's 
console immediately next to the surgeon’s ears, External monitors arouind the bedside also display the picture of 
the internal surgery which the surgeon is seeing within the console. So all members share a common view of the 
inside of the patient and can talk to each other as if they remained around the bedside within arm’s reach of each 
other, 
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To teach and reinforce team management and leadership for surgeons there have previously been video 
instructions and role playing scripts that walk through each of the skills which have been identified as essential 
for surgical teams. The video recordings present previously enacted situations which can contain both correct 
and incorrect activities that the surgeon/student can be evaluated on through questionnaires following the video. 
But the situations do not require interactive participation by the surgeon. Live role playing events allow the 
surgeon and all of the actors to experience multiple variations on the situation and to explore unique ideas which 
emerge in real time. However, these are extremely difficult to coordinate and host. The working schedules of 
surgeons, circulating nurses, surgical technicians, and anesthesiologists are very different. Each profession is 
guided by different certifying boards, departmental management, educational requirements, and working hours. 
Arranging for live events within a hospital or at a professional conference can be nearly impossible with real 
professionals. At some educational conferences, these events have been organized using hired actors for the 
members of the team. These remain expensive and rare events. Though these methods have proven useful, some 
of their limitations may be overcome through a computerized, interactive, game-based learning environment. 


This paper describes a project to create a surgeon leadership and team management virtual environment which 
could be used at a robotic surgeon’s leisure. This environment can include more variations in activities than can 
be easily captured in videos and can provide some of the richness of live role-playing events, but without the 
expense and logistical hurdles. 


This paper describes the process used to design, prototype, and field the virtual world application. The 
application is currently in final in-house testing and will be released for open community testing in the near 
future. After that it will become the basis for a validation trial focused on its educational effectiveness. 


BACKGROUND 


‘The robotic surgery team training virtual world (RoboTeamView) is the sixth product of a larger effort to create 
materials for the Fundamentals of Robotic Surgery (FRS) program, an authoritative, standardized curriculum for 
certifying the knowledge and skills of aspiring robotic surgeons (Smith, Patel, Satava, 2013), 


‘The FRS program has leveraged the expertise of more than 50 of the leading robotic surgeons in the world as 
well as a number of educational and engineering professionals, to develop materials which surgical educators 
can use to bring new surgeons to a common, measurable, and professionally accepted level of proficiency prior 
to performing surgery on human patients (see Figure 2). These materials include: 

a. Online Curriculum consisting of text, slides, photos, and videos for teaching the cognitive knowledge 
needed by robotic surgeons; 
Psychomotor Skills Device which measures the tactile skills of a surgeon using the robot; 
‘Team Training videos which convey material similar to that included in the RoboTeamView game; 
‘Team Training Role Playing Script which can be acted by live role-players; 
Intuitive Surgical da Vinci Skills Simulator (DVSS) exercises; 
Mimic dV-Trainer simulator exercises; 
Simbionix Robotix Mentor simulator exercises; and 
Robotic Surgeon Team Training Virtual World (RoboTeamView) for teaching team skills to a surgeon 
who is training alone. 


PRropaos 
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Figure 2. Fundamentals of Robotic Surgery Curriculum Products — (a) online curriculum, (b) 
psychomotor skills device, (c) team training videos, (d) role playing script, (e) Intuitive DVSS simulator, 
(f) Mimic dV-Trainer simulator, (g) Simbionix Robotix Mentor simulator, (h) RoboTeam View virtual 
world. 
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METHODOLOGY 
This project used the ADDIE process for design and development of the learning application (Branch, 2010), 
Analysis of Problem. 


Surgeons with years of experience in bedside surgery (open and laparoscopic) described a sense of separation 
from the operating team when they began using the robot for procedures. In spite of the video and audio assistive 
tools which allow members of the team to communicate with each other, the physical separation and lack of 
direct line-of-sight to the team allowed the surgeon to immerse himself in a private world during a procedure. 
Effective communication with the team became something that required a higher level of conscious effort to 
maintain throughout a procedure. Surgeons needed to learn when to use the communication tools in the robot 
and when to disengage from the robot in order to handle situations which required more direct human-to-human 
contact (Hanly et al, 2006). 


Analysis of Users. 


‘There are two primary users of this virtual world. The first are attending surgeons, fellows, and residents who 
aspire to practice robotic surgery. The second are experienced robotic surgeons who require additional training 
in working effectively with a team. Both groups have limited time to focus on new curricula beyond their current 
work load. Both must lear independently in an environment that they access themselves. They do not have 
dedicated classrooms, equipment, instructors, and class hours as do traditional university students. In most cases, 
the student is expected to learn on their own time and without the collaboration of other members of the OR 
team, 


Analysis of Environment. 


The users typically possess extensive medical and surgical skills, but very limited computer skills. They are 
typically not proficient at installing new applications on computers, or they are using machines that are 
controlled by corporate IT restrictions which prohibit unauthorized applications. These characteristics led to a 
focus on a web-based application with a plug-in which auto installs if needed, and which can be approved for use 
across the corporate environment. 


Design of Instruction. 


Instruction is based on the widely used TeamSTEPPS curriculum (Safny et al, 2011; Thomas and Galla, 2013) 
and WHO checklists for surgery (WHO, June 2008). This material is then modified for application in a robotic 
OR environment. The exchanges with team members in this environment are largely prescribed and standardized 
to reduce miscommunication and the omission of important steps. The instructions for the game were based on 
prior work to create role-playing scripts for robotic OR team members. 


Design of User Experience. 


‘The primary instructional environment is a virtual robotic operating room which is populated with four avatars 
representing the other members of the team. The surgeon is either viewing a surgical field inside of a patient or 
the team around the operating table. In the former case, the surgeon interacts with a surgical video using menu 
selections at key decision points. In the latter, the surgeon queries an avatar for information and gives it 
instructions to be followed. The primary goal of the environment is to lead the surgeon through specific 
scenarios and assist them in understanding the correct actions that they should apply. This is primarily a learning 
environment and secondarily an assessment tool. 


Development of Virtual Environment. 


Virtual Heroes has previously created a number of healthcare virtual worlds which included digital assets that 
appear in this virtual world. The essential new asset which had to be created was a 3D model of the da Vinci 
surgical robot, a complex piece of machinery with many visible pieces. The robotic arms and hand controls need 
minimal animations for these team scenarios. More work was required for the multiple menu items necessary to 
present all of the decision actions of the team. 


Development of Video Integration. 
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The project required the integration of 3D virtual world assets with prerecorded videos of the surgical field. 
These videos were drawn from the extensive video library of a leading robotic urologist and some videos were 
custom made during live surgeries. From these we were able to select segments of surgeries which corresponded 
to the lessons being taught in the virtual world. Synchronizing virtual actions with video events allowed us to 
avoid creating virtual representations of complex internal human anatomy and the manipulations of those 
models. 


Development of Evaluation Criteria. 


‘The scenarios provide multiple decision points at which a surgeon/student must select the correct response from 
a small list of options. The correct selection will lead to acceptance by the avatars and progression to the next 
step. An incorrect selection will cause the avatars to offer advice or to ask leading questions to guide the surgeon 
to a correct action. Performance evaluation is a summation of the correct and incorrect actions taken by the 
surgeon during each scenario. Benchmarking those scores will be part of a future validation process in which 
proficiency levels will be established based on the scores of expert and novice subject 


Implementation of Training Program. 


The training program will be implemented in multiple steps. Initially, the RoboTeamView will be made 
available to a small number of robotic surgeons who assisted with the development of the new curriculum. They 
will provide feedback during the early releases to assist in reprogramming or redesigning features of the 
application. The secondary release will be to a community of expert robotic surgeons who have contributed to 
the creation of previous FRS program materials. These experts are the conduits for sharing the application with 
aspiring robotic surgeons at multiple hospital systems and organizing a validation trial using surgeons, fellows, 
and residents. Finally, the application will be made publically available at no charge for access by anyone who is 
interested in using it for their own personal learning or as a tool within in an educational environment. 


Evaluation of Effectivene: 


Acceptance of this material by instructors and institutions for education in robotic surgeon training is an 
encouraging and valuable achievement. But it does not constitute scientific evidence of validity as an effective 
teaching tool. This will be achieved via a multi-site validation trial of the tool with the goal of demonstrating that 
it is an equal or better method of teaching team leadership skills than the existing methods. 


DEVELOPM. 


Data Acquisi 


The development process began with the acquisition of knowledge and data. The game development team 
observed multiple procedures in the robotic operating room. They were able to watch and listen to all of the 
activities that occurred, and to see each member’s role throughout a procedure. They also witnessed the 
transition of nursing support staff completing a shift or leaving for a break during a procedure. Following this 
exposure, robotic surgeons were interviewed, introduced to the product concept, and provided their guidance on 
how such a product could be structured for effective education. An analysis of the published literature of the use 
and availability of simulators or virtual worlds for robotic surgeons indicated that a leadership-focused tool for 
team communication skills had not previously been created (Kumar, Smith & Patel, 2015). Therefore, many of 
the educational design concepts of this project were being created for the first time. 


‘The team reviewed existing curriculum in textual script and video recording formats. These were based on best 
practices which have been created by the TeamSTEPPS program and the World Health Organization for safe 
communications in the operating room. Together with the data collected from the surgeons, the team arrived at a 
small set of scenarios to be included in the virtual world, as listed in Table 1 
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‘Table 1. Surgical Scenarios Created 


ST. Tastrument Exchange (Request and Call Back) 

$2. Material Insertion & Retrieval (Request and Call Back) 
$3. Two-challenge Rule for a Safety Issue (CUS and SBAR) 
4. Personnel Change (Handoff Responsibilities) 

5. Check Back 

6. Emergency Robotic Undocking Procedure 

s7. Pre-Brief (Checklist or Sign-in) 

$8. Post-procedure Debrief (Checklist or Sign-out) 

9. Recoverable Robotic System Fault 

$10. Non-recoverable Robotic System Fault 

SIL. Broken Instrument 

$12. Difficulty Removing/Reinserting an Instrument 

$13. Loss of Insufflation of Patient 


The game calls for a combination of 3D computer graphic assets and live surgical videos. Through the 
cooperation of several surgeons the project received access to an extensive library of thousands of surgical 
videos. These videos are all usable for educational purposes through signed releases from the patients. As 
specific scenarios and 3D actions were developed, the team located an existing surgical video with actions which 
corresponded to the scenario. Using such a large video library made it possible to avoid either video recording a 
mulated surgery or attempting to create a realistic virtual representation of all of the surgical activities. In spite 
of the size of this library, it was necessary to custom record some actions during surgeries for this project. The 
current level of simulation technology is challenged to graphically model human tissue with manipulation, 
dissection, and blood flow. Some surgical VR simulators contain very realistic, but limited representations of 
surgery which require significant computer hardware to run. For this reason this project relies on video recording 
to represent actions in the surgical field, which comes with some inherent limitations to interactivity. 


Us 


sr Experience 


Role Definition 

Early discussions within the development team and with surgeons were focused on who would be the training 
audience for the tool. Since there are five members of the OR team who must learn to work together, should this 
tool provide a user interface and curriculum for each of these as potential trainees? Such a flexible tool seemed 
possible since the scenario is the same for each role, only requiring the removal of one script to allow a human 
user to play that part. However, since there were no previous tools of this type to use as guidance, solving such a 
multifaceted problem could lead to confusion and delays that would threaten the success of the project. Also, 
achieving acceptance of the tool from five different sets of professional and certifying organizations seemed to 
be a much larger problem. Therefore, the design focused only on training the surgeon, as was done with previous 
curriculum products. But, the virtual world and other training products may become the basis for variants that 
are targeted at the circulating nurse, first assistant, surgical technician, or anesthesiologist in the future. Since the 
game creates a single-user domain, there is no need for computer servers to coordinate the interactions of 
multiple players within the same scenario. A single scenario can be served to any number of users 
multaneously, but each of these runs independently without the need for coordination between multiple 
players. 


Dual Domains 

During a procedure, the surgeon occupies two very different domains. One is as a member of the team that 
surrounds the operating table to address the patient from the outside. The other is a more private domain in 
which the surgeon is immersed within the internal anatomy of the patient with audio communication to the 
outside team (see Figure 3). In the scenarios which are to be represented (Table 1) it is most accurate for the 
surgeon to act within both of these domains, which requires creating a simulated environment of both. Previous 
training curriculum in video and script formats had presented the OR only from the external bedside view, while 
existing simulators provided only the internal view. This game is the first to include two very different domains 
in which the surgeon is learning. For some scenarios a surgeon remains immersed in the patient while 
responding to the team and giving direction. But for others, the surgeon needs to learn to disengage from the 
internal view in order to address a more important issue in the external OR. Learning which domain is most 
appropriate for the surgeon has become part of the training that is uniquely provided by this game. 
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(a) Virtual Operating Room (b) Video Surgical Field 


Figure 3. Virtual World Representation 


Session Independence & Progression 

‘When a surgeon proceeds through a scenario, their progress is stored on the local computer. This allows users to 
interrupt their progress through a scenario, but return to the same point when they pick-up the game at a future 
time. Information about progression is also exported to the Moodle Learning Management System (LMS) to 
provide scoring and evaluation of the players. When a surgeon chooses to terminate a scenario prior to 
completing it, the LMS has a record of progress that has been made. In future versions this information will 
make it possible for the surgeon to complete a scenario from multiple devices by loading past progress from the 
LMS. This capability is a potential extension should early users discover that it is an essential feature. 


Security 


Like most corporate environments, the hospital IT infrastructure is tightly controlled and monitored to protect 
again hostile external and internal actions. It also blocks certain private and social services which are not 
considered productive in a corporate environment. As a result, many ports and some data formats cannot be used 
by applications like this virtual world. 


‘The application was designed for Windows 7 and 8 operating systems and the Internet Explorer v.7+ browser 
because these are the most common within the hospital. Virtual Heroes bases many of their custom projects on 
the Unreal engine licensed from Epic Games for simulation projects. This engine and the game content are 
configured as a one-time browser plug-in to eliminate issues with asking users to perform multiple heavy 
downloads and installations. As a plug-in, this process is largely automated upon first use of the application. 
However, corporate IT restrictions still verify that the plug-in is permitted within the controlled hospital 
infrastructure. Therefore, the plug-in was treated as a new application which had to be reviewed for security and 
stability issues before being allowed to enter a hospital computer. 


Additionally, once installed, the plug-in communicates with the LMS via unique ports and data formats which 
had to be approved to traverse the hospital network (see Figure 4), 


The application was originally developed and shared from a Virtual Heroes server, and was then tested on 
personal computers on an open commercial network. Once a basically functional version existed, a hosting site 
‘on the internet was created which required a fresh install away from the Virtual Heroes machines. This 
demonstrated that the application was portable enough to be hosted on a customer's servers as opposed to the 
developer's servers. Finally, the hospital IT department created a hosting site within the hospital infrastructure, 
approved the plug-in on hospital computers, and opened the necessary communication ports for the application. 
The goal is to host the application on a site which can be accessed by surgeons both inside and outside of the 
hospital infrastructure. Robotic surgeons who are not employees of Florida Hospital will access the external site. 
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Figure 4. RoboTeamView System and Network Architecture 


User Evaluation 


‘The performance of the surgeon is evaluated as they interact with the scenarios and the dynamic avatars in the 
game. The application provides very direct guidance regarding the steps that are expected. The intention was for 
the game to be more of an educational environment than an assessment tool. The design allows surgeons to work 
through the scenarios without a human instructor and to learn the necessary information for performing as a team 
leader. There are numerous opportunities for a surgeon to make decisions, each of which is captured in the LMS 
to provide some measure of their performance. But, an attentive surgeon can learn the correct responses from the 
avatars without having to consult a human instructor. Therefore, the measurements of performance are actually a 
measure of the surgeon’s ability to learn and adapt to the guidance of the avatars in the game. 


Each surgeon logs into the system to create a record of on-going performance in the LMS. The Moodle LMS 
also provides login for an instructor who can access all student performance data. This allows a hospital, college, 
or education center to track the performance of their people and to insist on a specific level of mastery in 
association with credentialing, risk management, and educational progression. 


VALIDATION AND DEPLOYME! 


‘The virtual world application has been completed and is being evaluated by experienced robotic surgeons and 
teaching faculty at Florida Hospital. The feedback from these professionals will be incorporated into the 
application before releasing it to a larger audience for independent and objective validation trials. The FRS 
project has developed research relationships with a number of leading medical institutes around the world. These 
have participated in the validation of previous FRS products and have shown their ability to organize and 
conduct these types of trials. The sites listed in Table 2, as well as others who have shown interest in the 
materials, will be invited to access this application and participate in a multi 


Following these trials, the revised application will be made available on the TrainRobotic.com web site for 
aspiring robotic surgeons, instructors, and medical training facilities to use as a curriculum for training robotic 
surgeons in their leadership responsibilities within the OR. Users of the application will be able to track student 
performance via the linked LMS. 
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Table 2. Robotic Surgery Curriculum Validation Site List 


Florida Hospital Nicholson Center, Orlando FL | Lahey Health and Medical Center, Boston MA 

University of Athens Medical School, Greece Hartford Hospital, Boston MA 

Imperial College, London UK Louisiana State University School of Medicine, New 
EndoCAS, Pisa Italy Orleans 

Baylor University Medical Center, Dallas TX Madigan Army Medical Center, Seattle | WA 
Carolinas Healthcare System, Charlotte NC University of South Florida Health CAMLS, Tampa FL. 
Lehigh Valley Health Network, Allentown PA Methodist Medical Center MITIE, Houston TX 

Duke University Medical Center, Raleigh NC University of Pennsylvania Medical Center, Philadelphia 
CONCLUSIONS 


The primary goal of this project was to determine whether an effective leadership training application could be 
created for robotic surgeons who must learn to lead a team in the OR while performing surgery. The bulk of the 
efforts went into identifying which scenarios should be represented and how the information should be 
structured to create an effective training tool. The resulting product demonstrates that such an application can be 
created and that it satisfies potential users. As of this writing, the tool has not been used to train surgeons, 
fellows, or residents in OR team leadership. Neither has a validation trial been conducted to compare the 
effectiveness of this method against existing methods, e.g. didactic lectures, textual instructions, video recorded 
cases, and live role playing events. The next step is to conduct such a validation trial to determine whether the 
application is effective at teaching these skills to robotic surgeons. The results of these experiments and 
educational experiences are potential topics for future publications. 


Questions that remain outstanding include: 

‘* Will experts and instructors incorporate the application into their curriculum? 

‘* Do surgeons who use the application actually have better patient outcomes? 

‘+ Is the application better than or equal to existing methods of teaching these skills? 


‘* Is the product sustainable over a period of years, both financially and as educational content? 
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ABSTRACT 


Faced with an age of reliance on technology and innovative advances, surgeons are using cutting-edge 
robotic systems to perform complex procedures and virtual reality simulators for specialized skill training. 
The virtual environment and controllers in surgical simulators are reminiscent of those in videogames. So, 
can playing video games develop skills similar to those used in robotic surgery? 


This paper compares the performance of video gamers, medical students, and “lay people” to expert robotic 
surgeons on a robotic surgery simulator. Participants recruited from the UCF College of Medicine, UCF 
FIEA, and Florida Hospital completed a demographic questionnaire. The subjects then performed three 
computer-based perceptual tests and participated in two warm-up tasks on the Mimic dV-Trainer to 
familiarize themselves with the system. The experiment then measured their performance over eight trials 
of two core simulated exercises. After completing these trials, participants completed a post-questionnaire 
about their experience 


Analysis of the data did not verify differences between the groups for the perceptual tests except for the 
time to complete scores in the Flanker and subsidizing tasks, in which expert surgeons took significantly 
longer than other groups. Significant differences were found between the groups for the first and eighth 
trials of the simulated exercises, with surgeons performing better than other groups. All groups improved 
significantly from trial one to trial eight, with surgeons performing better than all groups. Gaming console 
type positively correlated with Overall Score in the Ring & Rail exercise, as well as Time and Economy of 
Motion in the suturing exercise. No other correlations were found 


The results are in contrast with prior literature on video game experience in laparoscopic surgery, 
suggesting that gaming abilities do not translate to all surgical modalities. Future research is necessary to 
further examine the impact alternative skillsets may have on surgical skills, 
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INTRODUCTION 


Surgery is generally described as fitting into one of two modalities—open and minimally invasive, the 
latter of which includes laparoscopic and robotic-assisted (i.e. robotic surgery) procedures. Robotic 
surgery, the most recent iteration of laparoscopy, typically implies that the surgeon’s movements are 
facilitated through a computer driven system to manipulate surgical tools. This field evolved from the 
prospect of surgeons performing life saving procedures on soldiers in combat zones from remote locations 
anywhere in the world, an application referred to as telesurgery. 


This concept has not completely come to fruition, however the fundamental research resulted in the 
commercial the daVinci Surgical System that is now used to perform everyday procedures in urologic, 
gynecologic, ENT, and general surgery specialties called (Barbash, Friedman, Glied, & Steiner, 2014; 
Serati et al., 2014; Maan, Gibbins, Al-Jabri, & D'Souza, 2012; Luca et al., 2013; Zureikat et al., 2013), The 
surgeon manipulates controllers at the surgeon console to manage up to four robotic arms, including a 
camera, attached to a separate patient cart. The camera provides true stereoscopic vision to the surgeon, 
facilitating a synthetic tactile sensation and depth perception. Attached to the other robotic arms are various 
instruments, which move in a similar manner as the surgeon’s hands (Figure 1). 


Patient Cart 


Master Controller 


Vision Tower 


igure 1. The daVinei System 
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While this system integrates robotics into medicine in a way that may seem more science fiction than 
reality, society is actually connecting with technology in unforeseen ways. Traditional surgical skills are 
being transcended by cutting-edge technologies, which require surgeons to possess distinct skill sets from 
those of the past and which overcome a learning curve to acquire the technical (i.e. psychomotor) skills, 
associated with using the daVinci system. Efforts have focused on developing specialized curricula for the 
training of such skills (e.g. the Fundamentals of Robotic Surgery and Robotic Training Network), but can 
learning curves be reduced to facilitate a faster acquisition of skills in surgical trainees? 


Previous research has established that trainees with video game experience demonstrate increased abilities 
on basic laparoscopic skill trainings (Rosenthal et al., 2011; Grantcharov, Bardram, Funch-Jensen & 
Rosenberg, 2003; Rosser et al., 2007). Also, video games have proven to be valuable training tools for 
basic laparoscopic skills (Rosser, Gentile, Hanigan, & Danner et al., 2012; Badurdeen et al., 2010; Ju, 
Chang, Buckley, & Wang, 2012; Bokhari et al. 2010; Schlickum, Hedman, Enochsson, Kjellin, & 
Fellander- Tsai, 2009; Giannotti et al., 2013). Certain genres of video games have established effects on 
perceptual skills similar to those required by robotic surgeons, yet few have attempted to make a 
connection between video game experience and robotic surgical skills (Green & Bavelier, 2012; Green & 
Bavelier, 2007; Chien et al., 2013; Harper et al., 2007). 


Thus, this research aims to examine the performance of experienced video gamers while using a robotic 
surgery simulator, and compare the performance of this population with experienced robotic surgeons, 
medical students, and laypeople. The purpose is to determine the effect that video game usage may have on 
the perceptual abilities that are used for robotic surgery. Contrary to previous research that used surgical 
trainees with minimal gaming experience, this research aimed to utilize subjects with high levels of gaming 
experience and compare their abilities to subjects with different levels of expertise. This study also looks at 
the groups’ ability to acquire basic surgical skills using the simulator. 


METHODS 


Recruitment 


Participants in this study included video game experts (VGEs), expert robotic surgeons, medical students, 
and “laypeople” (i.e. individuals without formal medical education or extensive gaming experience). VGEs 
were recruited from a local university offering degrees specializing in game design and development (i. 
Florida Interactive and Entertainment Academy [FIEA]). Potential VGE subjects were required to be 
enrolled in a game design program and self-report daily videogame play of at least two hours per day, five 
days per week. Expert robotic surgeons were recruited from Florida Hospital, Florida Hospital Nicholson 
Center training courses, and at relevant surgical conferences. These individuals were practicing physicians 
and self-report performing at least 100 robotic surgical procedures, of which he or she performed at least 
50% of the procedure on the surgical console. Medical students were recruited from the University of 
Central Florida College of Medicine (UCF CoM) and laypeople were recruited from all data collection 
sites. Potential subjects were excluded from the study in the case of having experience in more than one 
participant category (¢.g. a medical student or expert robotic surgeon who engages in regular gameplay of 
more than two hours per week) 


Materials 


Alll subjects completed a pre-questionnaire, which gathered demographic information (e.g., age, gender, 
handedness, hours of weekly gameplay, number of robotic cases). The participants then performed three 
computer-based perceptual tests: a Flanker compatibility task, a subsidizing task, and a Multiple Object 
Tracking (MOT) test. The Flanker compatibility test requires the participant to indicate the orientation of a 
single arrow in the center of a group of several other arrows. The arrows are randomly generated to all face 
the same orientation (congruent) or face the opposite direction of the target arrow in the center 
(incongruent). This tests attentional capacity by requiring the subject to focus solely on the relevant arrow 
and ignoring other stimuli, The subsidizing task also assesses attentional capacity by requiring subjects to 
identify the number of dots that appear on the screen by pressing the associated number key. In the MOT 
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task, users must track specific objects while they move across the screen with other identical objects, which 
assesses visual attention (Figure 2), 
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Figure 2. Examples of the Flanker, subsidizing, and MOT tasks 


Participants then performed two warm-up exercises on the Mimic dV-Trainer, Pick & Place and Basic 
Camera Targeting, to familiarize themselves with the system and system controls. All subjects then 
performed eight trials of two core exercises to test various basic skills (Table 1). Ring & Rail | and Suture 
Sponge | will serve as the primary exercises for data collection, After completing all exercises on the dV- 
Trainer, specific metrics are shown to the participants: Overall Score, Economy of Motion, Time to 
Complete, Excessive Instrument Force, Instruments Out of View, and Master Workspace Range. These 
primary metrics are exported for each exercise and used with other metrics to form the scoring system 


Table 1. dV-Trainer exercise descriptions 


Exercise Purpose Objective Skills Trained 
Warm-up Exercises 
Pick & Place | Introduction to using stereo vision | Place colored objects in Endowrist 
and EndoWrist instruments for | matching colored containers. | Manipulation 
picking up and placing objects, 
Basic Camera | Learn to accurately position the | Manipulate the camera to ‘amera Control 
Targeting camera while working in a large | position light blue sphere 
workspace while practicing to camera targets in the center 
keep the instruments in view and _| of your sereen’s dark blue 
developing stereo depth acuity. _| crosshairs 
Core Exercises 
Ring & Rail! | Coordinate control ofan object's | Pick up aring and guide the | Endowrist 
Position and orientation along a ring along a curved rail ‘manipulation, 
trajectory using the EndoWrist Camera Control 
instruments 
Basic Suture | Improve dexterity and accuracy | Insertand extract aneedle | Endowrist 
Sponge when driving a needle through a _| through several targets on the | manipulation, 
deformable object. edge of a sponge with amera 
random variations in their | Control, Needle 
positions. Control, Needle 
Driving 


After completing all trials, participants completed a post-questionnaire regarding their experience with the 
system (Figure 3) 
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Figure 3. Order of study procedures 

RESULTS 

Demographics 

Table 2 shows descriptive characteristics of the participants. Gamers indicated playing on average 11.71 
hours of video games per week and having 17.85 years of gaming experience. On average, expert robotic 
surgeons performed 503 total robotic cases and 127 cases per year. While none of the expert surgeons 
reported currently playing video games, 29% indicated playing video games in the past. Thirty-three 


percent of lay people also indicated playing video games in the past. 


Table 2. Descriptive Statistics 


Descriptive Statistics 

Gamers Medical Students Laypeople Experts 
nm 40 24 42 7 
Age 25.38 25.63 29.45 42 
Male 15% 70.83% 52.38% 71.43% 
Female 22.5% 29.17% 47.62% 28.57% 
Right Handed 87.50% 95.83% 83.33% 100% 
Left Handed 12.50% 417% 16.67% 0% 


Cognitive Tests 


For the Flanker and the subsidizing tasks, an ANOVA was performed to compare the four groups in terms 
of percent of correct responses and average response time (ms) for incongruent and congruent arrows. No 
statistical differences were found for the percent correct for the Flanker test, however completion times for 
the congruent and incongruent representations were significantly different between the groups (Congruent 
p<0.005; Incongruent p=0.007). Expert robotic surgeons took longer in both instances to perform the tasks. 


No significant differences were found for the percent correct on the subsidizing task for any groups. 
Similarly to the Flanker test, completion times were significantly different between the groups (p=0.001), 


with expert surgeons performing slower than the other groups. The MOT test was analyzed using a non- 
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parametric test to compare the number of correct responses. No significant differences were found for any 
groups for the MOT test. 


The cognitive scores were also analyzed in terms of certain demographic responses to determine if an 
association exists between the demographic characteristics and the cognitive test scores. A Pearson 
correlation coefficient was calculated. The characteristic of age positively correlated with the Flanker Time 
(p=0.008) and Flanker Incongruent Time (p<0.005). Age negatively correlated with the hours of weekly 
video game play (p=0.010). Age was also negatively correlated with the number of correct responses in the 
normal level of difficulty MOT task (p<0.001). 


Simulator Scores 


The simulator scores were analyzed in terms of three performance metrics for both simulated exercises: 
Overall Score, Economy of Motion, and Time to Complete. Overall Score is a composite score comprised 
of multiple performance metrics, including Economy of Motion and Time to Complete, Economy of 
motion is the total distance that the instrument tips moved and is measured in centimeters. Time to 
Complete is the total number of seconds required by the user to perform the exercise. 


An ANOVA was used to determine if differences existed between the groups for the first (i.e. Trial 1) and 
the last (ie. Trial 8) of the Ring & Rail I and Suture Sponge for the performance metrics. The groups 
performed significantly different for the performance metrics for trial 1 in both exercises except for the 
Overall Score of Ring & Rail. Using a Least Significant Difference Test, experts performed significantly 
better than other groups for the metrics. Similar results were found for trial 8 of both exercises. Experts 
again performed significantly better than all groups in trial 8 for both exercises scores all metrics (Table 3). 


Table 3. Differences in simulator scores for Trial 1 and Trial 8 


Trial 1 ‘Trial 8 
Ring & Rail 1 
‘Overall Score =0.086 ‘Overall Score p=0.256 
Economy of Motion (p=0.05 Economy of Motion =0.005 
Time (P=0.005 Time ‘P=0.005 
Suture Sponge 
‘Overall Score P=0.001 ‘Overall Score P=0.05, 
Economy of Motion p=0.005 Economy of Motion P=0.005 
Time (p=0.001 (p=0.001 


The simulator performances were also analyzed using an ANOVA to determine if differences exist between 
the groups in terms of the change in performance from trial | to trial 8 for both exercises separately. A 
difference existed in the average Overall Score and Economy of Motion metrics from trial | to trial 8 for all 
groups in the Ring & Rail | exercise (Overall Score p<0.001; Economy of Motion p<0.001). Experts were 
found to be significantly different from the other groups for both metrics (Overall Score p=0.045; Economy 
of Motion p=0.002). A significant interaction was found between the trials and the groups for the Time 
metrics (p=0.006). The main effects of the trials were not examined due to this interaction. 


A difference existed in the average Overall Score and the Economy of Motion metrics from trial 1 to trial 8 
for all groups in the Suture Sponge exercise (Overall score p<0.001; Economy of Motion p<0.001). Experts 
were also found to be significantly different from the other groups for both metrics (Overall Score p<0.001; 
Economy of Motion p<0.001). A significant interaction was found between the trials and the groups for the 
Time metric (p=0.011). The main effects of the trials were not examined due to this interaction. The 
average of each metric across the eight trials for each exercise can be seen in Figure 4 and Figure 5 
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Figure 4. Average scores for groups across cight trials of Ring & Rail 1 
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Figure 5. Average scores for groups across eight trials of Suture Sponge 


An analysis was conducted to determine if an association existed between the perceptual test scores and the 
simulator metrics for the two exercises. The Flanker scores for the percent of correct responses negatively 
correlated with time to complete for the Ring and Rail 1 exercise (p=0.006). This suggests that as the 
correct response percentage increased the time taken to complete the exercise decreased. No other Ring & 
Rail | metrics correlated with the perceptual tests. No associations were found between the Suture Sponge 
scores and the perceptual test scores. The subsidizing and MOT task scores were not significantly 
correlated with any metric values for Ring and Rail | or Suture Sponge. 


Video Games 


The video game experience of the subjects was also analyzed to determine if certain aspects of video game 
play were associated with simulation scores. For this analysis the type of game and console played by the 
subjects was used. The game type ranged from not using videogames, playing slow-paced strategy games 
(e.g. puzzle games), playing both types of games, or playing fast-paced action games (e.g, first person 
shooters). The console type ranged from not playing video games, using a controller with minimal hand 
movement (e.g. Playstation4), using all controller types, or using a controller that may require larger 
movements (e.g. Wii). 


No significant correlations were found between the type of video game or console played and the 
performance metrics for either exercise for trial 1. A significant positive correlation for Overall Score and 
the type of console was found for trial 8 of Ring and Rail | (p=0.049). This association suggests that as the 
movement to control the game increased, the Overall Score increased. A significant positive correlation 
was found between the type of console and Economy of Motion and Time for trial 8 of Suture Sponge 
(Economy of Motion p=0.044; Time to Complete p=0.002). This suggests that as the movement to control 
the video game increased, the time to complete and the distance traveled by the instrument tips increased 
(ie. slower and less efficient with movements). 


DISCUSSION 
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The assumption that video gamers will perform better than others using a virtual reality robotic surgery 
simulator is very common. The manipulation of the hand controls and the users interaction with the 
synthetic environment seem comparable to that of a video game. Contrary to these similarities and prior 
literature in laparoscopy, video gamers in this study did not perform better than other groups including the 
“Average Joe” in a robotic surgery simulator. The results did suggest that subjects who use higher 
movement game controllers (ic. Ninetendo Wii) scored higher in the Ring & Rail | exercise. However, 
those individuals also took longer and were less efficient with their movements in the Suturing exercise. 


The results from this study align with the few studies that have examined the impact of video game play on 
robotic surgical skills. Chien et al. (2013) found that in comparison to a group using task specific virtual 
reality training, a control group using video game training did not perform as well on an actual task using 
the surgical robot. The authors also found that using a video game to train actually had a negative impact 
on the post-training performance. Harper et al. (2007) found that video game players tied significantly 
fewer knots using the surgical robot and also suggest that video games may have a negative impact on 
surgical skills, 


Why does prior video game experience impact basic laparoscopic skills, but not robotic? Differences may 
be contributed to the distinctness of the systems that the users are interacting. The skills developed in two- 
dimension video games may transfer more appropriately to laparoscopic surgery, which uses a two- 
dimensional screen, as opposed to the three-dimensional view in robotics. Laparoscopy involves 
contrasting movements to the primarily fine motor movements of robotic surgery and it is possible that 
gamers are more inclined with the manual dexterity associated with laparoscopy. 


While this study was unable to validate enhanced abilities of video gamers in robotic surgery, the results 
demonstrated that the effect video game play has on surgical skills is nuanced by the surgical technique. In 
a technologically dependent society where video games have become an integral past time, this analysis of 
skills will likely become more valuable as other fields leverage the gaming generation’s experience into 
training. The findings can be generalized to domains outside of medicine utilizing robotic and computer- 
controlled systems (e.g. unmanned vehicle operation), speaking to the scope of the gamers’ abilities and 
pointing to the capacity within these systems. 


Future research should examine the impact alternative skillsets may have on a user’s abilities in a robotic 
surgery system (e.g. playing sports). The gamers in this study did not perform significantly better than lay 
people, which may imply that other factors or hobbies contributed to the performance. Only one surgical 
robot currently exists, however others have realized the technological advances and future iterations of 
surgical robotic systems are imminent. As these new technologies enter the market, it will be critical to 
evaluate how these skillsets may be valuable to the field of robotic surgery 
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ABSTRACT 


Complexity in surgery lies at the crossroad between the complexity of the human body and the capabilities of the 
surgical tools available. While we continue to improve our understanding of the body, we are also inventing new 
tools to address and correct issues through surgery. As a result, the complexity of surgery is expanding on two 
fronts simultaneously. This creates a lifetime learning environment for practitioners and a challenge for the 
systems which educate, measure, certify, regulate, and privilege surgeons. The models of training in this field are 
slow to evolve and still rest on a foundation of lecture and hands-on practice which has changed little in 100 
years, Surgeons largely believe that real hands-on practice with human tissue ~ excised organs, cadavers, and 
live patients — is the most effective form of training. But it is also the most expensive, difficult to facilitate, and 
least accessible form. 


The emergence and maturation of the concept of blended learning in public and military education may prove 
equally valuable in surgical education and training. Creating a learner-centric environment in which multiple 
modes of education are encouraged, available, integrated, and accredited can potentially increase the level of 
competence of new surgeons, maintain competence in practicing surgeons, and provide objective metrics to the 
public and hospital systems. 


This paper defines a framework for blended surgical training using principles developed for the military. This, 
framework includes knowledge and skills-based training in both an individual and a group learning environment 
which includes distance and e-learning sessions, face-to-face engagements, laboratory events, and operating 
room experiences as modes of surgical education that are not integrated into a coherent program with defined 
metrics. The goal of the framework is to apply blended learning principles to the surgical education and training 
community, with reference to prior activities in public and military education. 


Keywords: blended learning, learning science, surgical training, medical education 
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NATURE OF SURGICAL EDUCATION 


Developing the ability to perform surgical procedures requires mastering complexity in multiple dimensions. It 
calls for cognitive knowledge, psychomotor skills, and team management techniques. Mature attending surgeons, 
must be competent in all of these to lead a surgical team through a procedure in which a patient’s health and life 
are at risk. Accomplishing all of this has led to extensively long educational programs which typically include a 
four year bachelor degree, a four year medical degree, a three to five year residency, and perhaps a two year 
fellowship. But in many countries, safety regulations have limited the number of hours that a resident or fellow 
can learn and practice during this process (Funke, 2013). At the same time, the explosion of medical tools and 
technologies has led to ever increasing numbers of available procedures and the specialized knowledge that, 
accompanies these. Acquiring more knowledge and skills by increasing the number of practice hours is not 
possible, so one alternative is to improve the methods of education, hopefully increasing the speed at which 
mastery can be attained through more efficient educational methods. 


Beyond the formal educational programs, practicing surgeons are also constantly acquiring new skills through 
continuing medical education (CME) courses. These provide knowledge, skills, and team management for a new 
procedure, tool, or technique in a one or two day event. The degree to which this material is mastered is based, 
not just on the quality of the instructor and the intelligence of the student, but also on the instructional methods 
that are applied and the tools which are used to transfer knowledge and skills. Instructors for these courses, 
typically prepare traditional slide presentations, videos of procedures and tools, and hands-on laboratory sessions, 
with tools and tissue. Assessment of performance and skill acquisition is usually based on subjective instructor 
observation and quantitative scores on exams. But these courses typically lack an objective metric for the 
acquisition of psychomotor or team management skills. Educators search for additional tools and methods that 
can provide accurate assessments of performance in these compressed events. 


Medical education from CME to under graduate medical school has always attempted to blend books, hands-on 
labs, apprenticeships, and research (Cooke, 2010). This model has been created, but its implementation is 
different at every institution based on faculty abilities, facilities available, patient presentation, scheduling, and 
other factors. This results in a wide variation of effectiveness and the emergence of a rumor-based reputation of 
the quality of each program. Nissen (2015) has recently published his view that CME in particular is not 
effective at improving surgeon performance or patient outcomes. He maintains that the CME system of 
education as a whole needs to be reformed, a major criticism of the current practice. 


BLENDED LEARNING 


‘While medical and surgical educators search for effective teaching methods and tools, the public schools have 
been actively developing concepts labeled as “blended learning” (Bonk, 2006; Horn and Straker, 2015). In its 
current form, this focuses on the integration of in-class, lecture-based instruction with electronic, online, 
independent educational materials. Horn and Straker (2015) have explicitly identified four dominant models of 
blended learning that are in use in America’s classrooms. These include — the rotation model, flex model, a la 
carte model, and enriched virtual model. Each of these represents a different sequencing or emphasis of learning 
via teacher-led instruction, collaborative activities, and online instruction. Given the estimated $60 billion 
investment in computers for classrooms (Christensen, 2011) the emphasis has been on how to add value 
equivalent to this enormous investment in basic technology. 
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Popular definitions of blended learning focus on this public school environment. 


“Blended learning is a formal education program in which a student learns at least in part through 
delivery of content and instruction via digital and online media with some element of student control 
over time, place, path, or pace. While still attending a ‘brick-and-mortar’ school structure, face-to-face 
classroom methods are combined with computer-mediated activities.” (Horn and Straker, 2015) 


This is certainly one important environment. But the structure and limitations of the K-12 classroom are not 
necessarily appropriate for other environments. Throughout these two books there is an emphasis on control of 
immature students and assessment via tests which indicate whether the student should progress. But there is little 
consideration for adult learning in which assessment can take more interactive and dynamic forms, such as 
demonstrated performance of skills. McCafferty and Desaulniers (2004), Fautua et al (2014), and Schatz et al 
(2015) have all explored what blended learning can contribute to military training programs. Their work and 
lessons learned appear to be much more similar to adult medical and surgical training than the K-12 educational 
texts on blended learning. 


But a more appropriate and useful definition is something similar to that quoted from Michael Orey in 
McCafferty (2004): 


“Blended learning is the ability to choose among all available facilities, technology, media, and 
materials matching those that apply to my prior knowledge and style of learning as I deem appropriate 
to achieve instructional goals.” 


This definition is much more encompassing of all learning tools and methods available. It also seems to capture 
an active role on the part of the student, rather than treating the student as a passive agent to be acted upon by the 
teaching system. Peder Jacobsen cites a US Department of Labor study which estimates that 70% or more of all 
learning occurs on the job, not in a structured classroom or educational environment. This means that traditional 
educators and formal education materials are only able to impact 30% of the learning experience. This calls for 
both effective methods for impacting that limited volume of experience and a search for methods which can 
become part of the majority 70% that is happening in other environments (Saltzman, 2010). 


Military training organizations have adopted the terms “Live, Virtual, and Constructive” to refer to different 
modes, objectives, and methods of simulation-based training. These terms came into popular use in the late 
1990s to describe the literal situation that existed at that time. Training events could be defined as purely Live, 
or Virtual, or Constructive, lacking the technology and experience to integrate any two of these together. Today 
the terms have merged into the “LVC” acronym which indicates that we have arrived at a state of blended 
training in which all three are generally included in any simulation-based training event of significant size. This 
evolution has relied on improvements in computer and networking technology, but also on the expertise of 
scientists and engineers who have developed an understanding of how to bring these together. Accomplishing 
this in an educationally effective manner has been a different and slower process to emerge. The blending of 
technologies together may have been accomplished without an understanding of how the resulting product would 
be used, so lacking capabilities that would be essential for effective learning events. Similar challenges exist 
when attempting to blend simulation into medical education (Gardner, 2015). As a result, educational designers 
are often faced with a simulation federation which is not well designed to support the needs of the training and 
education mission. Fautua et al (2014) offered ten recommendations, some of which can be applied to 
engineering the technology in support of blended learning objectives (Table 1). 


Table 1, Fautua’s Recommendations for Effective Blended Learning 
Provide the reason why for goal-oriented adult learners. 

Gain commitment to the training by trainees and their leaders. 

Create content which can be reapplied to sustainment training. 

Focus on the staff being trained, not the canned event. 

Embed diagnostics which can guide other learning modes. 

Schedule and plan for remediation. 

Train the trainers to apply blended concepts. 

Collect feedback throughout the blend. 

Emphasize human-system integration. 

0. Shift the culture of trainees, trainers, and leaders toward blended approaches. 


BeRNP Maen 
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Each of these represents a very valuable lesson for other subject domains, such as medicine and surgery, and 
were applied in the development of a blended training framework for surgeons, as presented in this paper. This, 
leads to the need for a larger definition for the term, something more akin to: 


“Learning in the 21" century, especially with adults, requires leveraging all of the information delivery, 
communication, data collection, and assessment tools that are available. Blended learning may be a 
useful term to discuss, design, and evaluate learning methods that are very different from those of 
previous generations, even to the extent of eliminating face-to-face sessions and expert instructors. 
Perhaps the greatest value in the term ‘blended learning’ is not in defining it, but in allowing it to 
undefine the historical approach to education.” (Original derivation by the authors for surgical course 
design.) 


SURGICAL BLENDED LEARNING 


Just as previous works have applied and expanded blended learning concepts to military education and training, 
this paper extends those ideas to the field of surgical education. Organizations which offer adult, surgical, 
continuing education wrestle with many of the same challenges and opportunities as the K-12 and military 
education systems. But, time is a driving concern when trying to separate practicing surgeons from their daily 
duties with patients, staff, and hospital administration. Little time is available and each hour that is allocated 
must provide a return greater than the good which could have been performed in the direct delivery of care. 


Adult surgical training programs, often referred to as Continuing Medical Education (CME), generally focus on 
the hands-on psychomotor skills acquisition through the use of new tools, new techniques, and realistic tissue 
models, These are excellent learning and practice environments which can be produced only in specialized 
facilities for a limited number of participants. Blending this core event with other technology-enabled modes of 
education could potentially deliver knowledge and skills more smoothly, while also improving the learning 
density of the high-end core event, 


Figure 1 illustrates two of the most common pathways for CME in surgery. The traditional path (T) begins when 
students physically arrive at the training location and assemble as a group in front of the faculty member. This, 
very familiar path is dominated by lecture and typically incorporates photos and vivid videos of surgical 
performance. It is typified by a one-way flow of information and knowledge. The second step in this path 
incorporates hands-on laboratory practice with tissue models. The initial focus is on knowledge, which then 
flows into applying that knowledge to develop specific skills. Students are expected to apply both of these in 
their surgical practice when leaving the course, 
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Figure 1. Traditional (T) and Extended (E) Surgical Learning Path for CME 
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Courses of this type are typically offered in a condensed single or multi-day format, Surgeon-students in most 
countries can afford to allocate only a few days to attend such a course and are eager to carry what they learn 
back to their practice. 


The second extended (E) pathway enables learners to familiarize themselves with knowledge materials prior to 
physical assembly at the course, This occurs through online, web-based, internet enabled materials and could 
rightly be seen as equivalent to the traditional definition of blended learning bridging the internet and the 
classroom, This mode is expected to either reduce the amount of group time required for the lecture phase of the 
physical class, or seen as an opportunity to reinforce information by presenting it multiple times in different 
formats to improve retention, Currently, few or no courses prescribe an independent curriculum of skills 
exercises prior to physical attendance at a course, though there are opportunities to add this modality. 


Experience with hundreds of surgical CME courses indicates that there remains a great deal of opportunity to 
further extend and enhance the learning pathways which can be applied effectively. Blending methods for 
surgeons who must learn knowledge, skills, and teamwork calls for a broader view of the blended term and a less 
linear and standardized version of training for the surgeons. The model shown in Figure 2 illustrates the 
opportunities that exist with the tools and technologies that are available and experience with directing students 
through an enriched form of education and training, 
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Figure 2. Enriched Blended Learning Environment for Surgical CME 


Within this model the two dominant existing learning pathways can still be seen. But they have been enriched 
internally and extended externally to take advantage of tools and technologies that currently exist, but which are 
seldom integrated into the learning experience. 


Independent-Knowledge 


This model begins with independent knowledge acquisition in which each student accesses introductory and 
case-based materials on their own, typically prior to group events. There are hundreds of online, web-based, 
case-based courses available today. One huge library of these which focuses on surgical videos can be found at 
the European WebSurg.com site maintained by IRCAD (Institut de Recherche contre les Cancers de IAppareil 
Digestif) in Strasbourg, France. Dozens of training centers, medical device companies, and universities maintain 
similar, but smaller, online libraries which can be blended into a specific, custom surgical curriculum, These are 
attractive resources because many medical educators believe that e-learning leads to faster knowledge 
acquisition and better retention rates (Ruiz, 2006). 


More recently, there have been a few game-based learning tools which focus on surgery. These provide a more 
interactive and dynamic experience which can respond to the learner's actions and provide corrective guidance. 


2016 Paper No, 16024 Page 6ofl0 


Interservice/Industry Training, Simulation, and Education Conference (VITSEC) 2016 


One such tool is the Fundamentals of Robotic Surgery Team Trainer (www.trainrobotic.com) which was 
designed to guide a surgeon through TeamSTEPPS procedures in a robotic operating room environment. 


Independent-Skills 


The independent acquisition of basic skills prior to group instruction is not typically part of an instructional 
curriculum, However, using a number of existing, low cost “box trainers” and inexpensive disposable 
instruments a surgical student can practice maneuvers which are demonstrated in online course materials. 
Through centuries of surgical education, a model for the substitution of various grocery meats, excised organs, 
and animals has evolved to replace practice on live humans. These materials are recognized as acceptable 
learning environments and many of them are readily available to any learner. More recently, this model has been 
augmented with an array of synthetic materials which are available from vendors or can be manufactured from 
silicon compounds. 


Telemedicine sessions can also be conducted in which an instructor remotely views the performance of the 
student and provides customized feedback. Cellphones and laptops are all equipped with the necessary hardware 
and software to enable this remote, but individualized, instruction. However, this has not yet been integrated into 
surgical education programs. This mode has the potential to make a significant contribution to student skills 
prior to group assembly, just as online material is used to build a base level of knowledge prior to a traditional 
course. 


Group-Knowledge 


Traditional courses have been very heavy with the dissemination of knowledge materials in a group setting via 
lecture and videos of actual surgeries. These remain an important part of a blended curriculum because they 
allow synchronous discussions between students and instructors. But these may also be supplemented with team 
training role play and table top exercises. 


Group learning has typically assumed that all of the participants are physically assembled together for the event. 
But with the advent of online virtual worlds, it is also possible for group interactions to be facilitated via 
computer networks. F2F group training may be supplemented with virtual worlds, or the team training may 
occur prior to the physical meeting, as is accomplished with tools like 3DiTeams 
(http://www. virtualheroes.com/portfolio/Medical/3DiTeams). 


Group-Skills 


Surgical training programs are currently most heavily focused on a group setting for skills development. This 
mode is primarily accomplished with wet and dry lab exercises (biologic and synthetic materials), But it could 
also include the use of new virtual reality simulators, OR observation, and apprenticeships with an experienced 
surgeon. 


Blending the curriculum between simulators and wet/dry labs has proven to be more difficult than it would first 
appear. Surgical simulators are faced with computational limitations on representing the behavior of soft tissue 
and blood flow. These soft features are unique to the human body and have not been previously addressed by 
either the military simulation or the video game visualization communities. Those domains have made 
significant improvements in the visualization and physical behaviors of hard, rigid objects like vehicles, terrain, 
and weapons. But they have done litle in the area of tissue flexibility. Even soft objects like clothing and hair 
have been modeled as rigid objects, rather than tackling the soft fluid behavior problems. Given this situation, 
surgical simulators typically focus on exercises which develop specific motor skills with simple puzzle 
environments — e.g. pegs, rings, rails, and simple sponges. This has led to exercises which develop basic, 
beginner-level skills, but which do not represent the realistic human tissue and blood flow that are necessary for 
models of real procedures. Funke et al (2013) and other authors maintain that simulators are valuable for 
younger students, but not for more experienced practitioners who need real patients, partly for this reason, This, 
means that the simulators offer an experience that is similar to simple box trainers, but with more accurate and 
automated metrics collection. Blending these into a curriculum is difficult to justify when evaluated via a 
cost/benefits analysis. VR simulators are just beginning to include soft tissue models of procedures with 
bleeding. As these improve, there will be a unique place for them in many curricula. 


OR observation and apprenticeship are a very attractive and popular part of a surgical course. Learning surgeons 
are eager to see an experienced instructor performing real procedures with human patients and explaining the 
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challenges and rational for each step. This level of realism is difficult to reproduce in wet/dry labs or simulators, 
so represents a unique learning opportunity which includes real time discussions with the instructor. 


Using these four categories as the framework, blending learning content across all available modes yields a 
surgical education framework as shown in Table 2. 


Table 2: Blended Learning Framework with Specific Surgical Content 


Tndependent Knowledge 


‘Web Course Introductory knowledge about diagnosis, medical indications, instruments, OR resources, 
approach to anatomy, legal regulations, and costs. Explain the rich environment and 
history in which the instrument or procedure exists. Create a common knowledgebase 
before group events. 


Web Video Tniroduce the common use of the instrument or the procedure, Present multiple variations 
and complications that can arise. Organize complications into categories with a shared set 
of solutions. 

Web Game Create an interactive case-based problem for diagnosis, decision making, and an 


appreciation for the complexity of the environment and the procedure, 


Tndependent Skills 


White Box Prescribe skills exercises which can be accomplished independently with accessible and 
affordable devices and tissue. Encourage rehearsal of common approaches and skills. 
Demonstrate self-guided education pathways for use in training residents. 


Telemedicine One-on-one remote evaluation of white box skills, Personalized direction on improvement 
and collection of pre-course metrics for improvement. 


‘Group Knowledge 


F2F Discussion Prior independent activities transform this mode from one-way lecture to two-way 
discussion. Less focus on basic knowledge and more attention to subtle details and 
potential solution approaches. 


Video Evalu 


Evaluation of video cases as an educated cohort of professionals, rather than one-way 
explanation of basic features of the case. Highlight situations which will be presented in 
later group skills sessions. 


Team Training Role playing for basic procedures and extended complications, Table top exercises on 
resource management and interactions with other departments. 


Group Skills 

Simulation VR Hands-on experience with skills exercises and automated metrics collection, contributing 
to learning curve improvements. Pre-qualification for use of real instruments and tissue. 

Dry Lab Focus on capabilities and features of instruments, approaches to tissue, planned 
intervention on patient. 

Wet lab ‘Application of instruments and approach to biological tissue (animal and cadaveric). 


Focus on response of real tissue to previously learned use of instruments, Environment for 
some complications and unexpected responses. 


OR Observation Observation of real environment with experienced instructor. Interactive discussion on 
strategy, expectation, and process. Opportunity for complication and resolution. 


‘Apprenticeship Partial participation in real environment. Opportunity to perform as part of an experienced 
team, Open to guidance and criticism by instructor surgeon. 


LEARNING METRICS 


The technologies that make rich and effective blended learning possible also offer the power of accurate and 
automated metric collection and computation. Schatz et al (2015) emphasized that blended curricula and systems 
should create a learner-centric model of education which is data driven and ubiquitously accessible. Technology 
and blending techniques can transform the industrial model of uniform education into something that is more 
customized with more objective metrics of performance. 


‘When implementing blended learning programs at IBM, Lewis and Orton (in Bonk and Graham, 2006) reported 
that evidence of effective programs was determined by (a) student reactions to the experience, (b) measured 
learning that occurred, (c) evidence of transfer to practice, (d) measurable business impact, and (e) a positive and 
significant return on investment (ROI). 


The metrics of greatest interest in surgical training are almost identical to these. Student reaction to the 
experience is important for repeatability of the program when surgeons have multiple options for learning the 
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material, as opposed to mandatory programs where the learner cannot select out. Formative and summative 
assessments are typically part of the educational program to provide feedback to the hospitals or surgical boards 
that support or prescribe the training. Some of these metrics for a blended program are described in more detail 
below. Evidence of transfer to practice has been a universally difficult metric to collect. Most courses send 
surgeons a survey between one and two months post-course asking for their feedback on how the training has 
impacted their practice of medicine. These are problematic because most surveys are not completed and returned 
(anecdotal rates are 5-10%), and those that are represent self-reported application without objective data to 
support the claims. Physician practices and hospital departments do have some ability to measure the business 
impact of courses which enable new procedures or the use of new devices. These can be measured by the 
number of patients who have been treated using the new skills and knowledge. Though objectively measurable, 
this information is typically considered business proprietary and is not shared with the external organizations 
that provide the training. The medical community would expand the scope of this metric to include “improved 
patient outcomes” as a measurable business impact which is important to society. Using this same data, practices 
and hospitals are able to calculate the ROI for new procedures and devices, and this information remains private 
within the business unit as well. However, business impact and ROI can be inferred when the same organization 
repeatedly sends surgeons to a program. 


Instructional designers are generally most interested in metrics which can be extracted during the educational 
process as evidence that learning is occurring, as opposed to the other meta-categories given by Lewis and 
Orton. Some typical metrics which can be collected are shown in Table 3. One purpose and advantage of 
blended learning is that performance can be improved through multiple learning modalities and learning metrics 
can extend across multiple modalities to provide a better measure of performance than when confined to a single 
mode. To illustrate this, the table is structured to show metrics which can be shared in common within a 
category 


‘Table 3: Blended Learning Metrics 


Tndependent Knowledge 


‘Web Course Number of correct/incorrect responses 0 test questions interspersed within web pages, 
‘Web Video video, and game vignettes, Efficiency through a game’s leaming path and selection of 
‘Web Game correct/incorrect branches in the course, video, or game. 

Independent Skills 

White Box Collection or observation of materials showing incisions, suturing, knots, etc. One-on-one 
Telemedicine remote evaluation of white box technique. 

Group Knowledge 

F2F Discussion | Interactive demonstration of knowledge and understanding. Accuracy of role playing and 
Video Eval efficiency of resource usage. 

Team Training 

Group Skills 

Simulation VR Digital metrics for efficient hand movement, errors, instrument collisions, blood loss, etc. 
Dry Lab Human observation of same metrics in labs and apprenticeship. Collection of synthetic 

[Wet lab tissue materials showing incisions, suturing, knots, etc. Damage to biologic and synthetic 

[OR Observation | tissue, blood loss. Discussions with surgeon to demonstrate understanding. Observed and 

[Apprenticeship | guided technique in apprenticeship, 


CONCLUSION 


Surgeon leaders like Richard Satava (2009) and Ajit Sachdeva et al (2016) have been promoting the addition of 
simulation devices to surgical training for a decade or more, along with the integration or blending of this tool 
existing modes of training. These efforts have had a noticeable impact on surgical training in specific areas 
(Gardner et al, 2015). Details on how to implement the integration of simulators have been proposed by various 
training centers around the country that are associated with the American College of Surgeons. 


Most students and instructors have experienced courses in which the age and relevance of the material are in 
need of revision. Over time content and presentation standards change. Outdated material threatens to provide 
both incorrect educational materials and an experience that is viewed with disdain because of its visible age. 
Blending multiple modes and devices into a single learning experience is a challenging endeavor. But, once 
accomplished, the difficult work is not finished. There remains the challenge of preparing an instructor to lead 
students through the blended curriculum, installing and training the technical staff that will build and maintain 
the electronic devices and information resources which enable the course, planning for the sustainability of the 
program over time in terms of cost and staffing, and planning for technology refresh as the devices and materials, 
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age. Kern and Hughes (2009) and McCaffery and Desaulniers (2004) both emphasize the need to keep pace with 
changes that occur in the teaching and simulation tools and in the knowledge and skills that are being presented, 
Creating complex blended courses brings with it the added effort and cost of updating the materials across all 
modes in order to maintain the relevance of the course. Organizations seeking to leverage the immediate 
advantages of multi-mode blended learning should be aware of the long-term commitment to investing more 
money and time to keep the material relevant, 


Creating a blended surgeon course is not just a project which can be accomplished in the near term and used 
indefinitely. A course which touches on all of the modes shown in the framework presented here may be 
effective and impressive for a period of a few years, but will eventually need a substantial investment of time, 
money, and expertise to maintain its level of educational effectiveness. 


REFERENCES 


Bonk, C.J., & Graham, C.R. (2006). The handbook of blended learning environments: Global perspectives, local 
designs. San Francisco: Jossey-Bass/Pfeiffer. 

Christensen, C.M., Hom, M.B., & Johnson, C.W. (2011). Disrupting class: How disruptive innovation will 
change the way the world learns. New York: McGraw Hill. 

Cooke, M., Irby, D., & Obrien, B. (2010). Educating physicians: A call for reform of medical school and 
residency. San Francisco: Jossey-Bass Publishers. 

Fautua, D. T., Schatz, S., Reitz, E., & Bockelman, P. (2014). Institutionalizing blended learning into joint 
training: A case study and 10 recommendations. In Proceedings of the Interservice/Industry Training, 
Simulation and Education Conference (I/ITSEC), Orlando, FL.. 

Funke, K., et al. (2013). Blended learning in surgery using the Inmedea simulator. Archives of Surgery, 398(2), 
335-40. 

Gardner, A.K., Scott, D.J., Pedowitz, R.A., Sweet, R.M., Feins, R.H., Deutsch, E.S., Sachdeva, A.K. (2015). 
Simulation-based surgical education: Best practices across surgical specialties relating to simulation- 
based training. Surgery, 158(5):1395-402. 

Horn, M.B., & Straker, H. (2015). Blended: Using disruptive innovation to improve schools. San Francisco: 
Jossey-Bass. 

Kern, D., Thomas, P., Hughes, M. (2009). Curriculum development for medical education. Baltimore: Johns 
Hopkins University Press. 

McCafferty, L.R., & Desaulniers, J.E. (2004). The revolution of blended training: The distributed instructor. In 
Proceedings of the Interservice/Industry Training, Simulation and Education Conference (V/ITSEC), 
Orlando, FL. 

Nissen, S.E. (20015). Reforming the continuing medical education system. Journal of the American Medical 
Association, 313(18): 1813-4. 

Ruiz, J.G., Mintzer, M.J., Leipzig, R.M. (2006). The Impact of e-learning in medical education, Academic 
Medicine, 81(3), 207-12. 

Sachdeva, A.K., Blair, P.G., & Lupi, L.K.. (2016). Education and training to address specific needs during the 
career progression of surgeons. Surgical Clinic of North America, 26(1): 115-28. 

Saltzman, P., editor. (2010). Best practices in surgical education: Innovations in skills training. Cincinnati, OH: 
Ethicon Endo-surgery Inc. 

Satava, R.M. (2009). The revolution in medical education-the role of simulation. Journal of Graduate Medical 
Education, 1(2):172-5. 

Schatz, S., Fautua, D. T., Stodd,J., & Reitz, E. (2015). The Changing face of military learning. In Proceedings of 
the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Orlando, FL. 


2016 Paper No. 16024 Page 100f10 


Interservice/Industry Training, Simulation, and Education Conference (VITSEC) 2016 


Surgical Simulator Showdown 


Danielle Julian, Alyssa Tanaka, Patricia Mattingly, Roger Smith 
Florida Hospital Nicholson Center 
Celebration, FL 
Danielle. Julian@flhosp.org, Alyssa. Tanaka@flhosp.org, 
Patricia Mattingly @flhosp.org, Roger.Smith@flhosp.org 


ABSTRACT 


The introduction of robotic technology in minimally-invasive surgery created a need to develop more efficient and 
effective training and assessment tools. Virtual reality simulators were introduced to the field to address this need. 
Currently, there are four da Vinci simulators - the dVSS, dV-Trainer, RoSS, and the RobotiX Mentor. These 
simulators offer basic training for novice robotic surgeons, familiarizing them with the skills needed to perform safe 
surgery. While there is literature available for each simulator individually, it can be difficult for a user to select the 
appropriate system to meet their training needs. 


Thus, this paper presents the results of a comparative analysis of the system components of each device (e.g 
exercises, scoring metrics, physical dimensions, and student management). Previous research has directly compared 
three of the four simulators, however this is the first study to compare all four. To collect the information, the team 
reviewed the device manuals for details on each system, contacted device company representatives, and explored the 
system capabilities firsthand, 


While all systems offer basic skill training in highly immersive 3D environments, each device offers unique 
advantages and capabilities for training robotic surgeons. The dVSS creates a high-fidelity training environment by 
leveraging the real robotic surgeon’s console for visualization and control inputs. The dV-Trainer, RoSS, and 
RobotiX Mentor offer simulated versions of these systems. Each includes system management services for 
instructors to collect, export, and analyze trainee scores. All systems have been the subject of multiple published 
validation studies, however these reports do not provide essential details on the nuances of each simulator. The 
analysis in this paper can be used to aid potential users, buyers, and trainers in identifying the features, which are 
more essential to their training centers. 
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INTRODUCTION 


‘With an increase in the number of minimally-invasive procedures, surgical education has shifted away from the 
traditional apprenticeship model (i.e., “See one, do one, teach one”) towards an experiential framework. Surgical 
trainees may encounter their first surgical experience on an inanimate training model, excised tissue, or as of more 
recently, a virtual reality (VR) simulator. These hands-on modalities provide a trainee with the opportunity to 
become familiar with equipment and instruments, develop skills (e.g., improved dexterity), and increase the 
understanding of surgical techniques and procedures (Polavarapu, Kulaylat, Sun, & Hamed, 2013). 


VR simulation was first introduced to surgical education in the late 1980s (Satava, 1993). Since implementation, VR 
simulators have been established as valuable training tools for the acquisition of basic surgical skills. They allow a 
trainee to safely overcome the learning curve associated with new techniques while providing independent and 
repetitive exposure in a safe and cost-efficient environment (Chou & Handa, 2006). The application of VR 
simulators in surgery has proven to be essential with the development and implementation of new technology and 
complex devices. 


One such device, Intuitive’s da Vinci Robotic Surgical System, introduces unique components not available in 
traditional surgical techniques. This system provides surgeons with 3D vision, 7 degrees of freedom of laparoscopic 
instruments, tremor damping, motion amplification, camera stability, and other advanced features (Palep, 2009). In a 
robotic procedure, the surgeon sits at the surgical console separate from the other surgical team members and 
patient. From this console the surgeon manipulates master controllers, which translate the surgeon’s movements into 
the smaller, more precise movements of the robotic instruments that are attached to a separate patient cart. The 
surgeon also controls the camera functionality using these master controllers. The camera provides magnified, 
stereoscopic vision allowing for depth perception and creation of a synthetic tactile sensation (Figure 1.). 


While this system offers multiple benefits, it introduces a technological divide between the surgeon and patient, 
which can lead to usability factors and a need for a specialized skillset. Providing a trainee with experience on the 
actual da Vinci system can be difficult due to the associated costs and resources required. Hospitals must make a 
large capital investment when adopting a robotics program and subsequently must recoup their investment via 
robotic procedures in the operating room. This often limits access to the system for training to time outside of 
normal operating room working hours. Along with accessibility limitations, training with the actual system requires 
the use of Intuitive’s surgical instruments, which incur an additional cost, 
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Figure 1, The da Vinci System 


Over the last ten years several VR robotic simulators for the da Vinci have become available for educational and 
training purposes (Figure 2). Currently the commercially available simulation systems are: 

+ da Vinci Skills Simulator (Intuitive Surgical Inc., Sunnyvale, CA); 

+ dV-Trainer (Mimic Technologies, Inc., Seattle, WA); 

+ Robotic Surgical Simulator (Simulated Surgical Skills LLC, Williamsville, NY); and 

+ RobotiX Mentor (Simbionix USA Inc., Cleveland, OH). 


da Vine Skis Simulator aVTrner Ress ‘Robotix Mentor 


ES 1 


Figure 2. Simulators of the da Vinci Surgical Robot 


Most hospitals or training centers will typically invest in only one type of robotic simulator. In general the systems 
are very similar, however each offers unique capabilities that may make it difficult for an institution to decide on 
which system is most appropriate for their specific training needs. Thus, there is a need for comparative evaluations 
of these simulators to aid potential buyers and users in selecting an appropriate device for their purposes. 


The objective of this paper is two-fold: to demonstrate a framework for comparing multiple training systems and 
demonstrate its use by providing readers with an objective comparison of the available VR robotic surgical 
simulators, This comparative framework provides potential trainers, buyers, users, and instructors with the 
appropriate information and details needed when considering an investment in an educational training device. This 
process can be leveraged across various training fields when a comparison of multiple training systems is needed, 


This is an extension of a previous analysis, which examined the functionality of only three of the simulators (ie., da 
Vinci Skills Simulator, dV-Trainer, and Robotic Surgical Simulator) and illustrated the capabilities side-by-side 
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(Smith, Truong, & Perez, 2014). Since the previous analysis, new technologies, exercises, and simulators have 
emerged. This paper provides comparative data on the functionality of the four commercially available robotic 
simulators. 


METHODOLOGY 


Before conducting the comparative analysis, the Florida Hospital Nicholson Center research department composed a 
catalog of the minimum system requirements needed in a robotic simulator to effectively train surgeons. These 
included hardware and software components, as well as aspects critical for using the system as an education and 
training tool. These components were identified using expert judgment on the critical aspects of the system that a 
novice surgeon would need to learn including system components (e.g. controls) and surgical skills that they would 
need to master (e.g. needle driving). Educational components were also identified to identify what components are 
necessary for a simulation system that is being used for education, The identified requirements from the actual da 
Vinci system were used as criteria for each simulation systems (Table 1). These requirements were compared across 
simulation systems and were used to communicate the accuracy of the simulators features to the actual system (i.e., 


the realism of the simulators) (Table 2).. 


Table 1. Selected Criterion for Simul 


lator Comparison 


Griterion ‘Meaning Purpose 
Hardware Components 
‘Aecessibility | Availability of simulator. The system may be a ] Provides surgeons with a convenient and 
stand-alone system (ability of operating | easily accessible training device. 
independently of the actual da Vinci surgical 
system) or embedded (utilizing the da Vinci 
surgical console). 
Ergonomics — | The ability to adjust the stereo viewer, foot Provides the surgeon with optimal 
pedals, and arm rest. positioning and maximum comfort. 
Master Naturally positioned manual manipulators Used to translate surgeons hand movements 
Controllers | attached to console within a fixed working into micro-movements of the instruments. 
space, 
Stereoscopic | The visual system used in the da Vinci surgical | Visual system provides surgeons with a 3D 
Visual system. This system provides two slightly perception to provide surgeons high 
System separate images. When these images are viewed | definition and natural colors. 
together it creates an impression of depth, 
‘Software Components 
Exercises Multiple levels of training scenarios for either | Provides training scenarios to educate user 
basic robotic skills (e.g,, suturing skills) or cn proper use of da Vinci surgical system 
procedural skills for specific robotic procedures. | and to provide repeat practice, while 
providing assessments of user’s 
performance. 
Scoring Established thresholds that provide users with | Benchmarks indicate acceptable and 
System scoring benchmarks set by expert robotic unsatisfactory scoring, Allowing the user or 
surgeons. administrator to track progression in each 
exercise. 
User ‘Allows user To create personal accounts, Also, | Personal accounts allow student account to 
Management | allows administrators or instructors to identit track and maintain training progress Provides 
and control the state of users. administrators or instructors with the ability 
to track and manage student accounts, 
Curricula ‘Ability for administrators to create and/or assign | To provide an optimal training based on 
Development | users a course of study and specified training. _| users experience and training needs. 
Data Export | Allows administrators or instructors to pull Exported data can be used to wack user's 
saved data (e.g,, exercise scores, attempts, etc.) | development and progression or can be used 
for a single student or an entire group. for statistical purposes. 
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From this analysis, the team identified multiple physical components of the actual da Vinci robotic system that must 
also be present in the simulation systems. That is, each simulation system should mimic major mechanisms of the da 
Vinci robot including: the master controllers, visual system, foot pedals, ergonomics, and size. 


The introductory skills required to use the robotic system are typically offered through basic exercises. These 
exercises are generally not clinically directed (je. require clinical decision making), but are focused on the 
mechanical and psychomotor skills required to drive the system. The number and types of exercises housed in the 
simulators were identified. For education and training purposes, it is also essential for the systems to have certain 
administrative capabilities relevant to the learner and the educator. Specifically, the scoring system, curricula 
development capabilities, user management functions, and data export process are critical for capturing and 
communicating learners’ performance. 


Table 2. Comparison of Simulator Features 


Simulator | Ergonomics | Controllers | 3D | Foot | Admin [Data | Scoring | Basic Total 
vision | pedals | control | export | System | Robotic | Score 
Skills 
Exercises 
DVSS + + + + a + + + +7 
av- + + + + + + + + +8 
Trainer 
Ross : + + + + B + + 76 
RobotiX. + 7 + + + + + + 47 
Mentor 


The team evaluated each system for these requirements by exploring the simulators firsthand, identifying the 
similarities and unique characteristics across the systems. We elected to purchase all of the systems to ensure that 
this evaluation remained objective and without undue influence from the manufacturers. The team also reviewed the 
device manuals to collect additional details about each system (Simbionix RobotiX Mentor User Guide, 2015; Skills 
Simulator for the da Vinci Si Surgical System, 2013; dV-Trainer Robotic Simulator User’s Manual, 2015; Robotic 
Surgery Simulator User's Manual, 2012). For further information, representatives of each of the manufacturing 
companies were contacted. 


RESULTS 


While all systems offer robotic skill training in highly immersive 3D environments, each device offers unique 
advantages and capabilities for training robotic surgeons. Each of these devices are manufactured by a different 
company and provides a unique hardware and software solution for training and surgical rehearsal. The sections 
below describe the different physical and software requirements of the four commercially available da Vinci robotic 
surgical simulators. 


Hardware 


Embedded vs Stand-Alone 
The majority of the available robotic simulators for the da Vinci system are customized stand-alone systems built to 
mimic the appearance and technical aspects of the actual da Vinci robot. However, the da Vinci Skills Simulator 
(VSS), also referred to as the “Backpack,” is a customized computer package that attaches to the actual surgical 
console through a single fiber optic network cable. Currently, there are two dVSS models, one for each da Vinci 
Surgical System model available on the market, the da Vinci Si and Si-e surgical system and the da Vinci Xi 
Surgical System, Each simulator model is only compatible with the corresponding da Vinci Surgical System. In 
other words, the dVSS Si model is only compatible with the da Vinci Si and Si-e surgical system and the dVSS Xi 
model is only compatible with the Xi surgical system. 


Attached simulators of this type are usually referred to as “embedded trainers” because they leverage equipment that 
has already been constructed, purchased, and installed for the use of the real system. Embedded trainers are popular 
in military facilities that may face limited space and weight constraints. These types of trainers significantly reduce 
the hardware needed solely for training purposes. The U.S. Navy uses embedded simulators aboard ships to reduce 
weight and space requirements, allowing them to train while the ship is at sea. In addition to saving space, these 
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trainers allow trainees to use the actual controls from the real system to operate the simulator. This type of training 
provides a realistic experience that is almost identical in feel to the actual system, which may contribute to higher 
transfer of skills. The dVSS allows the trainee to use the actual surgeon console and corresponding controls that they 
will use in a surgery, including the master controllers, visual system, and foot pedals. 


While embedded trainers offer many advantages, they also come with inherent disadvantages. The entire surgical 
system can be used for surgery without the simulator however, the simulator relies on access to the surgeon console. 
Therefore, if the surgeon console is being used in surgery, the simulator cannot be used. The surgical system is 
expensive and hospitals typically try to maximize the use in the operating room to recoup the investment, In a 
hospital with a high-volume of robotic surgery program that doesn't have a system dedicated for training, the 
accessibility and availability of the simulator may be limited. 


In addition to availability, embedded trainers increase the amount of use on the actual system, These types of 
simulators put more usage hours on real controls leading to increased maintenance costs for those devices. That is, 
heavy use of the dVSS comes with equivalent use of the actual surgeon console, which may ultimately lead to the 
need for more frequent maintenance. Most maintenance costs are covered by the hospital purchased warranty for the 
robot, so additional maintenance is typically not a financial cost, but rather impacts the availability of the system for 
surgical procedures. 


The remaining simulators, the dV-Trainer, Robotic Surgical Simulator (RoSS), and Robotix Mentor, are all stand- 
alone virtual reality simulators that mimic the hardware components of the da Vinci Surgical System. The dV- 
Trainer, RoSS, and Robotix Mentor are all designed to replicate the appearance of the robotic surgical console 
including, master controllers, visual system, and foot pedals. Unlike the dVSS, these systems provide training that is 
independent of the actual da Vinci Surgical System. This provides trainees with a more convenient and easily 
accessible training device. However, the disadvantage of these systems is that the hardware is simulated and does 
not exactly replicate the feel of the real robotic controls. 


Technical Components 

The dV-Trainer consists of three main pieces of equipment: a “Phantom” hood, foot pedals and a desktop computer. 
The hood replicates the stereo viewer and master controllers of the da Vinci surgeon console. The dV-Trainer foot 
pedals mimic those on the da Vinci surgeon console footswitch panel. This footswitch panel looks and functions 
almost identically to the robotic foot pedals. The high-performance desktop computer generates the 3D images and 
measures the movements of the master controllers. The dV-Trainer also leverages support equipment that includes a 
touch screen monitor, keyboard, and a mouse that enable an instructor to select exercises to build a curriculum for 
students and allow an administrator to manage the collected data, This simulator can be configured to imitate the S, 
Si, and Xi model of the da Vinci robot. 

The RoSS is designed as a single piece of hardware with a similar appearance to the robotic surgical console. The 
hardware device includes a single 3D computer monitor, commercial force feedback devices for hand controls, foot 
pedals that replicate either the S or the Si model of the da Vinci robot, and an external monitor for the instructor to 
view. This simulator can be configured to imitate either the S or the Si model of the da Vinci robot. 


The Robotix Mentor shares some similarities and differences with the dV-Trainer and RoSS. This system is 
composed of two mobile carts supporting the hardware equipment, One cart provides the replicated surgeon console 
with Sony 3D stereoscopic glasses, custom free-floating hand controls, and foot pedals. The surgeon console, 
controls, and vision system are mimicked in hardware, while a 3D software model replicates the functions of the 
robotic arms and surgical space. The second cart is connected and contains the high performance graphics computer, 
monitor, keyboard, and mouse. These components allow instructors to build custom curricula from the available 
exercises and manage collected data. 


Ergonomics 

It’s important that training systems provide users with an accurate ergonomic experience as compared to the real 
system or device. A simulation should not provide an artificially less or more comfortable experience, nor one which 
is less efficient and responsive than the real system (Smith, 2012). Within these guidelines, user preference and 
comfort are important considerations for any system. If a user feels as if the simulator is not equivalent to the real 
system, they may reject its value, become frustrated, and be hesitant to use it. The trainees may also become 
accustomed to the simulated features and lack knowledge and confidence with the real system. A major benefit to 
surgeons using the da Vinci surgical system is the improvement in ergonomic characteristics in comparison to 
traditional surgical techniques (Lux, Marshall, Erturk, & Joseph, 2010). The stereo viewer, foot pedals, and arm rest 
can be adjusted on the surgeon console, providing the user with maximum comfort and optimal positioning. Thus, it 
is important for the training systems to mimic the adjustable ergonomic components. 
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Since it is an embedded trainer, the dVSS allows learners to train in in the exact ergonomic positioning that they 
‘would perform surgery. The dV-Trainer provides a custom adjustable table, stereo viewer, and arm rest. While the 
system's hardware differs from that of the dVSS, it still provides users with the same ergonomic settings. 


Only the stereoscopic viewer can be adjusted on the RoSS, which may make it challenging for trainees of differing 
height to achieve optimal positioning. The RobotiX Mentor allows the trainee to adjust the arm rest, foot pedals, and 
the stereo viewer, but does not allow the user to change the height of the simulated console, 


Master Controllers 

The master controllers are the manual manipulators surgeons use to control the instruments and camera, The 
controllers on the actual da Vinci surgical system are attached to the surgeon console within a fixed working space. 
On either side of the master controllers are finger clutch buttons. These buttons allow the surgeon to adjust the 
positioning within the working space to prevent collisions of the master controllers and return the controllers to an 
ergonomically comfortable position. This functionality can also be accessed via a pedal on the footswitch panel. 
Each training system should provide a work space similar to the actual da Vinci system, a clutch button capability 
on both the master controllers and the footswitch panel, and mimic the usability components of the da Vinci's 
master controls, 

All of the simulators provide a mimicked version of the da Vinci Surgical System’s hand controls, including a finger 
clutch component. However, the actual system provides clutch buttons on both sides of the master controllers, an 
aspect leveraged by the dVSS. However, the standalone trainers (i.e. dV-Trainer, RoSS, and RobotiX Mentor) only 
provide the clutch button on one side of their controllers, Additionally, the appearance and usability aspects of the 
master controls differ significantly across all the simulators (Figure 2). 


‘da Vinei Skil Simulator ‘av-Trainer Ross, Robotix Mentor 


Figure 2, Simulator Hand Controls 


The dV-Trainer has unique controls which connect to three cables that measure movement, rather than the more 
precise master controllers that are used in the da Vinci robot. The RoSS uses modified SensAble Omni Phantom™, 
force feedback, 3D space controllers (3D Systems Inc., Rock Hill, SC). These devices have a much smaller range of 
motion than the master controllers on the da Vinci robot and therefore require more frequent clutching than the 
actual robot. The RobotiX Mentor utilize innovative free-floating hand controls that are tethered to the arm rest by a 
bundled electronic cable. The attachment and orientation of the hand controls were designed to minimize the 
interference and weighted drag on hand movements. However, these controllers and mimicked console provide a 
much larger working space than the da Vinci surgical system, therefore trainees can perform movements in the 
RobotiX Mentor that cannot be replicated in the actual system, 


Such high fidelity instrumentation provides the user with realistic controls in order to raise the level of immersion 
(Sherman & Craig, 2002). For example, military simulators have demonstrated this in the driving controllers of the 
Close Combat Tactical Trainer (CCTT) (Johnson, Mastaglio, & Peterson, 1993) and the recoil of simulated rifles in 
the small arms Engagement Skills Trainer (EST) (Platte & Powers, 2008). For the CCTT armored vehicle trainer, a 
great deal of effort was put into creating simulated driving controls which accurately mimicked the real vehicle, but 
which had longer operational life necessary for a training device (Johnson et al., 1993). When developing a surgical 
simulator the tactile fidelity of the hand controllers is significantly more important and of much higher resolution of 
control, 


Visual system 
The visual system in the da Vinci Surgical System provides the surgeons with a true stereoscopic image. The 
endoscope (i.e the camera inserted into the abdomen) records the visual scene simultaneously with two lenses. The 
images are transmitted to the user's left and right eye, creating one seamless image in the stereo viewer. Therefore 
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each simulator system was assessed on their ability to mimic a stereoscopic, simulated 3D visual system closest to 
the real da Vinci surgical system, Each robotic surgical simulators should provide a 3D environment through a 
mimicked stereo viewer to provide a similar training experience to that of the actual da Vinci robot. As an embedded 
trainer, the dVSS leverages this hardware to create a high-definition, 3D virtual environment. The dV-Trainer, RoSS 
and RobotiX Mentor provide simulated technology and hardware to provide a similar experience. The dV-Trainer’s 
visual system uses a similar system to the actual robot: individual images for the left and right eye are transmitted 
from the computer and into the stereo viewer. The RoSS uses a single 3D computer monitor, built into the system 
and polarized glasses to produce 3D images, producing a visual scene with less depth of field than the actual da 
Vinci system. The visual system used in the RobotiX Mentor is also much different than the actual da Vinci surgical 
console. This viewer uses off-the-shelf Sony 3D stereoscopic glasses. The Sony glasses must be adjusted and 
focused for each use. To ensure optimal focus and vision, the user must maintain their body position used when 
originally focusing the viewer. Often, if the user moves the glasses must be readjusted for clear vision. 


Vision is often the primary sense used to immerse users into a virtual training experience (Sherman & Craig, 2002). 
It is imperative to present users with a graphic display that provides visual stimulus and delivers an immersive 
display, while accurately representing the training material, Slater et al (2009) demonstrated that subjects in a 
dynamic virtual environment which was created by ray tracing experienced a measurably higher level of immersion 
and response stress than those who experienced the same environment rendered via the less realistic ray casting 
method. 


Software 


Exercise Modules 

The exercise modules in each simulator are organized into hierarchical menus according to the surgical skill being 
addressed and the complexity of the exercise. To ensure effective training there are multiple core skills and relevant 
tasks that each simulation device should provide within the exercises (Table 3). Each simulator provides on-system 
instructions for every exercise in the form of textual documents and narrated video demonstrations. 


Table 3. Comparative Simulator Exercise Categories 


‘dvss ‘dV-Trainer Ross RobotiX Mentor 
Surgeon Console ‘Surgeon Console Orientation Module | Basic Robotic Skills and 
Overview Overview Motor Skills Tasks: 
Endowrist Manipulation 1 | Endowrist Manipulation | Basic Surgical Skills | Robotic Suturing 
Camera and Clutching | Camera and Clutching Intermediate Surgical | Robotic Single-Site 
Endowrist Manipulation 2 | Energy and Dissection Skills Suturing 
Energy and Dissection _| Needle Control Hands-on Surgical Stapler 
Needle Control Needle Driving Training Robotic Essential Skills 
Needle Driving Troubleshooting Procedural Modules: 
Games Games Hysterectomy 
Suturing Skills Suturing Skills Prostatectomy 

Lobectomy 


‘As described earlier, the dVSS Backpack is available in two different models which match the model of the robot to 
which it will be connected, either the Si (or Si-e) or the Xi model. There are differences in these two models of the 
simulator which should be identified. The dVSS Si contains 41 exercises organized into nine categories. Six of the 
41 exercises are from the Fundamentals of Robotic Surgery (FRS), a robotic surgical skills education, training, and 
assessment program for novice robotic surgeons (Smith, Patel, Satava, 2014). These FRS exercises are also available 
in the dV-Trainer and the RobotiX Mentor. The remaining exercises provide training on many crucial technical 
skills required for robotic surgery, such as needle control and suturing skills. The Xi introduces 13 new exercises for 
a total of 47 exercises organized into eight categories. The Xi offers new games and exercises that teach the use of 
advanced instruments (i.e., stapler) which is a surgical skill not addressed by previous simulators. The other 
exercises expand on training for camera control, endowrist manipulation and needle driving. Many of the original 
exercises have improved graphics from recent updates created by Mimic Technologies (Skills Simulator for the da 
Vinci Xi Surgical System, 2015) and new exercises have been added from a third vendor, SenseGraphics AB 
(Stockholm,Sweden). 
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Mimic Technologies and SimbioniX have developed many of the simulation exercises found in both the dVSS Si 
and Xi. As a result, many of the exercises in the dVSS and the dV-Trainer are similar. However, the current version 
3.3 of the dV-Trainer has a number of new exercises, which are not found in the dVSS, and the graphics have been 
upgraded so the visual presentation is no longer identical. This version of the dV-Trainer contains 65 exercises 
organized into ten categories. This count includes preview exercises for Maestro AR, described below, and FRS 
exercises described earlier. 


Mimic’s Maestro AR (Augmented Reality) provides procedure-specific exercises that allow 3-D interaction between 
the trainee’s virtual robotic instruments and real surgical videos. Maestro AR was designed to train procedure- 
specific anatomy, procedural steps, and decision-making skills. All of the exercises give instruction and guidance to 
help trainees identify anatomy and improve their technical skills including grasping, retracting, cutting, and energy 
application. In addition, the exercises test a trainee's knowledge of the procedural steps with multiple choice 
questions during the exercise (dV-Trainer Robotic Simulator User’s Manual, 2015; Kumar, Smith, & Patel, 2015). 


The RoSS simulator contains 52 unique exercises, organized into 5 categories, and arranged from introductory to 
more advanced. The RoSS system has fewer exercises but most include three levels of difficulty where each level is 
actually a unique exercise, This company has developed a set of 3-D exercises that are unique from those found in 
other simulators. They also provide optional video-based surgical exercises, called Hands-on Surgical Training 
(HoST) modules, in which the user is guided through the movements necessary to complete an actual surgical 
procedure. These guided videos leverage the force feedback capabilities of the hand controllers to push and pull the 
student’s hands to follow the simulated instruments on the screen. They require the student to perform specific 
movements accurately during the video before the operation will proceed. 


The RobotiX Mentor organizes its’ 53 exercises into eight modules which fall under one of two categories: Basic 
Skills or Procedural Modules. Thirty exercises fall under Basic Skills. Twenty three exercises fall under Procedural 
Modules which include complete procedures and procedure-specific exercises. Procedural exercises can be 
performed in a guided or unguided fashion. The guided version of the exercises prompts the user for each step of the 
surgery or task. In addition, the RobotiX Mentor has exercises that review anatomy and focus on team training. The 
RobotiX Mentor is the only simulator that offers complete surgeries and procedure-specific exercises in a fully 
simulated anatomical environment. The RoSS’ HoST and the dV-Trainer’s Maestro AR do have surgical exercises 
‘where the user performs procedure-specific tasks while actual surgical footage plays. However, neither of them offer 
those tasks in a fully simulated environment. Incorporating such exercises allows students to practice procedures in a 
safe and reproducible environment, while providing complications and emergent situations. 


Scoring System 

Upon completion of each exercise, all of the simulators automatically proceed to a scoreboard showing the student’s 
performance on the exercise. All four simulators use the host computer to collect data on the performance of the 
student in multiple performance areas (i.e. metrics). Using this data, the simulator provides scores for various 
surgical skills and a total composite score to signifying the user’s overall performance on the exercise. In addition to 
the metrics 

collected by the computer, the manufacturers of each simulator have created accompanying thresholds to indicate 
whether the student is attaining a specified level of proficiency for individual metrics and overall for each exercise. 
Al four systems have identified threshold scores to indicate acceptable and unsatisfactory scoring levels. The 
thresholds were developed based on the performance of experienced robotic surgeons. These are commonly 
interpreted as “passing” and “failing” (i.e. above acceptable threshold and below unsatisfactory threshold 
respectively), with a “warning” area between the two levels. Together these create green, yellow, and red 
performance areas to visually communicate the quality of the student’s performance on each metric 


All of the simulator manufacturers worked with experienced robotic surgeons to assist in establishing the relative 
values of each measure used in the composite score, just as they did for the threshold levels described earlier. 
Because these evaluations are the opinions of the specific people who collaborated with the company on the 
development of the system, the dV-Trainer, the RoSS, and the RobotiX Mentor provide the ability for a system 
administrator to adjust these levels to meet the needs of unique curriculum, courses, and students. However the 
AVSS is a closed, turnkey system which does not allow for threshold adjustments found in the other simulators. 


Each of the four simulators provide a different scoring system. The dVSS uses the Classic System, which represents 
the trainees score via a percentage of combined pre-established metrics. The dV-Trainer recently shifted from t 
scoring system and now provides the users score as a composite of total points earned, rather than percentages. TI 
scoring system is known as the Proficiency Scoring System. For this system, the instructor can change proficiency 
baselines and customize the scoring protocol to fit their needs. The RoSS scoring system follows similar principles 
of the dVSS and dV-Trainer’s scoring systems, however the scoring system is visually communicated differently 
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from the rest. The display presents a horizontal bar, which is colored green, yellow, or red to indicate passing or 
failing. The magnitude of the bar is a rough measure of the quality of performance (Figure 3), Additional displays 
show the numeric score and its relative position to a passing threshold, 


The RobotiX Mentor scoring metrics vary between simulation modules and cases. Some metrics apply to every 
exercise while others are only used for exercises in which they are relevant. Due to the novelty of the RobotiX 
Mentor, proficiency benchmarks have not been set for all exercises yet. Similar to the RoSS’ visual representation, 
exercises with defined proficiency levels provide a horizontal bar colored green, yellow, and red with a marker 
indicating passing or failing (Figure 3). For benchmarked scores, additional displays showing numeric scores can 
also be accessed, Exercises without set benchmarks provide a variety of exercise specific metrics divided into 
categories which are provided in a list format. 


‘da Vinci Skills Simulator @V-Trainer Ross ‘RobotiX Mentor 


Figure 3. Example Scoreboards from Each Simulator 


Progression monitoring and performance measures are two important components that should be incorporated into 
all educational training systems (Jones, Hennessy, & Deutsh, 1985). Therefore, each scoring systems was compared 
by their ability to successfully collect key metrics, present thresholds, communicate the learner's performance, and 
provide progression monitoring, The system should provide meaningful feedback that the trainee can use to 
specifically target skills that need additional attention and improve future performance, While some scoring systems 
may translate easily to users, others systems may be less explicit. This may make it difficult for trainees to truly 
understand areas that need additional training, 


User management 
Each simulator allows an administrator or instructor to manage and organize student performance according to the 
unique login credentials of the student. Additionally, all systems have a “guest” account to make the system 
accessible to anyone, but without the ability to uniquely identify and track individual performance under that guest 
account. The dV-Trainer, RoSS, and RobotiX Mentor allow the administrator to create user accounts directly from 
the systems, while the dVSS requires an external PC using a software program provided by Intuitive. 


Curricula Development 
Once a user account and login credentials have been created, administrators using the dV-Trainer, RoSS, or RobotiX 
Mentor can create and assign curriculum, The curriculum within the dV-Trainer and Robotix Mentor allow 
administrators to organize exercises into different assignments or phases, For example, a curriculum may consist of 
a warm-up phase with easy exercises, pre-course evaluations, and post-course evaluations. These would appear as 
three separate sections within the curriculum, ‘The exercises in the RoSS simulator are organized into a hierarchical 
tree structure according to the skills being taught. An administrator for this system can assign a specific branch 
within this structure as the curriculum for a specific user. But it is not possible to reorganize any set of exercises 
from multiple branches into a custom curriculum as it is in the dV-Trainer and RobotiX Mentor. 


The Robotix Mentor also provides administrators with the ability to add accompanying didactic material (e.g., PDF 
or video) into a curriculum folder, such as, video of real surgeries that are being simulated. In addition, this 
simulator includes an administrative management system, MentorLearn, which allows administrators to create, 
maintain, and assign specific curricula to specific users remotely. The dVSS does not provide administrators the 
ability to create or assign curriculum to users. 


Data Export 
Al four systems allow administrators to export data as a delimited file directly from the simulators. The dV-Trainer 
and RobotiX Mentor administrators can export data for a single student or an entire group. Further, these system 
allow administrators to export data according to multiple criteria, including, all of the data on the machine, or 
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subsets defined by the unique user ID, date range, completion status, or a specific exercise. The dVSS and RoSS 
administrators can export data files for each student account. While all systems render delimited files that can be 
removed from these systems for analysis, each system allow administrators to collect student data via different 
criteria, Some administrators may need to export data through a more sensitive criteria than a student account. The 
dV-Trainer and RobotiX mentor provide multiple ways to collect student data in comparison to the dVSS and RoSS. 


DISCUSSION 


Robotic surgery requires a unique set of surgical skills compared to other surgical techniques. Surgical training 
devices that can provide automated, objective performance assessment is desirable and useful for proficiency-based 
training. Robotic surgery simulators provide entry-level familiarization and skill development in a safe environment 
outside of the operating room. 


The simulators described in this paper are complex system that are valuable training tools at a lower cost. Each 
device offers unique capabilities for training robotic surgeons. While each simulator generally provides a physical 
environment conducive to introducing user to the robotic system, the simulated hardware varies across systems and 
can be different than the real-world equipment, Learners may experience a trade-off between lower price and perfect 
accuracy of a simulator. For example, the dVSS allows user to interact with the actual da Vinci surgical console and 
its features, however this device requires a greater investment than the other stand-alone simulators. 


Each system's software components provide trainees with the ability to practice basic robotic skills. Recent 
advancements in technology has introduced more procedural specific exercises to train an integration of multiple 
robotic skills and techniques. While all of the simulated devices provide core skills for novice robotic surgeons, 
more experienced surgeons may benefit most from the RobotiX Mentor’s procedure-specific exercises. In general, 
each system has some type of learning management system that educators can use to create curricula and track 
users’ performance. The dV-Trainer, RoSS, and Robotx Mentor provide multiple options for creating, customizing, 
and building curricula to provide optimal training. 


Unfortunately, due to limited accessibility to each of these systems, potential stakeholders may not have the 
‘opportunity to experience each of the simulation systems firsthand. However, it is imperative that hospitals critically 
evaluate the capabilities of training systems and how those capabilities align with their training needs prior to 
purchasing and incorporating a system into their training curricula, The goal of this analysis is to provide potential 
users, buyers, instructors, and trainers who have a need for a robotic surgery simulator with the information to make 
an informed decision on systems that are appropriate for their needs. 


It may be difficult for buyers to properly evaluate which system will meet their desired training and educational 
needs. Yet, if an organization or training center invests in a system that does not meet the learner and instructor 
needs or does not meet the environmental constraints, the system will not be valuable to the organization, 
underutilized, and lead to a decrease in return on investment. The process demonstrated in this paper can be 
leveraged into other domains, when multiple training systems exist in the market. 


Prior to purchasing a simulation system, there are several critical components an investor should consider. For an 
education and training device, it is important to evaluate potential systems for appropriateness to the learning 
environment, Other factors should also be considered including, whether the system is self-guided or requires 
management, durability, portability, and the ability to appropriately modify system to meet learning objectives. This 
process and results of this study will be used to inform future use of the simulation systems at the Nicholson Center. 
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ABSTRACT 


Does “validity” refer to the quality of an assessment, reliability of simulator outputs, or accuracy of internal simulation 
models? This question emerges in medical simulation and training, as educational, clinical, and engineering communities 
intersect. Each has developed a validation approach to meet their needs, without clear understanding of the other 
perspectives. Historically, validity has been assessed using a classical framework of content, criterion, and construct validity, 
concluding that a simulator is or is not valid. Validity has evolved into a unitary concept of construct, consisting of five 
distinct sources: content, response process, internal structure, relation to other variables, and consequences. Evidence for each 
source supports a score interpretation for a specific population, under a specific use case. This does not indicate that the 
nent itself is generally valid, much less whether the simulator can be relied upon to deliver accurate results, 


asses 


This unitary framework was adopted by the American Psychological Association as the standard for validating assessments 
and was recently endorsed as the “gold standard” for validating training tools. While this framework is effective for 
evaluating the appropriateness of an assessment, it may not be as robust for evaluating a simulation device used for 
assessment. This framework does not account for the physical and functional requirements of a physical system and the 
implications that discrepancies in those aspects may have on training and assessment. 


This paper compares the classical and unitary validity methodologies with a perspective on the application to training 
simulators, as well as examines the inherent limitations of both. Recommendations and industry standards from other fields 
are also examined for applicability to surgical simulation. Finally, a recommendation for the validity classification of surgical 
simulators is proposed. The future of surgical certification and licensing could be reliant on simulation, however validity 
standards must be established to support this goal 
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INTRODUCTION 


In simulation, many fields converge to create the specialized training tools used to provide learners with 
standardized environments for the safe acquisition of skills, relying on the expertise of engineers, educators, and 
subject-matter experts to create valuable training tools. It is imperative that these training systems are vetted to 
ensure that system performance meets the expected standards, a process typically referred to as validation. The 
resulting measure of validity refers to the degree to which a model or system is an accurate representation of the real 
world concept that it is intended to replicate (Sargent 2000, McDougall, 2007, AERA, 1997), 


The underlying validation process and associated implications are often subject to the field it is being referenced for. 
Using a flight simulator as an example, a computer programmer may validate the model in respect to how it 
performs against an actual system (e.g. aerodynamic characteristics). An engineer may assess whether the controls 
look and feel representative to the actual aircraft platform, and an educator validates that the flight assessment and 
After Action Review (AAR) accurately measure and provide relevant feedback on the trainee’s performance for a 
specific testing context 


The surgical field has adopted virtual reality (VR) simulators, similar to flight simulators, as a solution to limited 
training opportunities, regulated work hours, and a need for advanced training (Kuhn, 1962; Gallagher & Sullivan, 
2011). Similar to the validation of a flight simulator, each stakeholder involved in the development and 
implementation of a surgical simulator has a specific expectation for the concept of validity. The programmers are 
interested in how closely the physics models of the virtual environment are representative of the real world (e.g. how 
ue behaves when retracted) and the engineers verify that the controls function similarly to the actual surgical 
instruments. The educators and researchers are more concerned with how the benchmarks and scoring system 
translate to the learners 


The introduction of VR simulators coincided with a drive in the surgical field to move away from the traditional 
apprenticeship model and towards proficiency-based training. This has been critically important particularly in the 
specialized field of robotic surgery. Currently, four VR robotic surgery simulators exist. While all of these systems 
attempt to replicate the controls, visual system, and console of the actual surgical robot, each has unique qualities in 
regards to software, hardware, and assessment methods (Figure 1). This has resulted in many research studies 
attempting to validate these systems, as illustrated in a summary of these studies in Smith et al (2015) and 
Stephanidis (2015). 
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Figure 1. Different aspects of surgical simulation 


The validation studies that have been performed over the last decade have come at a time when medical education 
and assessment are shifting to new standards. Therefore, the interested educational communities have called for a 
shift away from the methods of previous studies and towards a new standard process. This discussion has revealed a 
distinct difference in the perspectives of different communities that are interested in the validation of simulators and 
of the educational outcomes they provide. In this paper, we present three dominant models for validation which may 
appear to be in conflict, but which actually represent the distinct needs of different communities, at different phases 
in a simulator’s lifecycle. This paper also provides a process for integrating multiple validation methods for 
effectively assessing educational technology. 


VALIDATION FRAMEWORKS 


Multiple professional communities have developed validation frameworks that address their own needs to insure, 
measure, and certify the accuracy, realism, and assessments provided by a simulator. The work of each of these 
communities is just beginning to be known to members of the other communities, which is triggering both mild and 
vehement disagreements about the meaning, purpose, and methods of validation. Cultural and intellectual clashes of 
these types have occurred repeatedly in other areas of science and engineering, Those cases, as in this, are often 
fueled by a lack of understanding of the perspectives and needs of the conflicting communities. 


In surgical simulation, several frameworks for proving validity have been proposed as the standard for validating 
educational technology. While the American Psychological Association (APA) endorses a “unitary” framework as 
the gold standard for validating assessment tools, this model alone does not account for the need to validate 
simulators from different perspectives in other fields, A shared understanding of all of the perspectives involved 
may eliminate much of the friction that is being generated in this area. The most prominent validation frameworks 
from three different communities is shown in Figure 2 and discussed below. 
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‘Student Assessment 


Requirements Verification Face Validity Response Process 

Conceptual Mode! Validation Content Validity Internal Structure 
Design Verification Construct Validity Relation to Other Variables 

Implementation Verification Concurrent Validity Consequences 
Results Validation Predictive Valicty 


Figure 2, Summary of the validation frameworks 
System Engineering Validation 


The community that develops simulators and implements a formal process for validating their accuracy and 
usefulness has relied on Sargent’s (2000) model for guidance through the engineering process, and indirectly the 
work of Balci (1997). In this model, the terms verification, validation, and accreditation (VV&A) are used to 
increase the preciseness of defining the steps in the process (Figure 3). However, this entire process is appropriately 
comparable to the other two frameworks that are explored in this paper. 


ite 
a7 See 


The creators and users of this framework are faced with a different set of 
problems than those who use of the other validation frameworks. Here, 
the emphasis is on guiding, controlling, modifying, and using a simulator 
as a hardware and software system or device. Because simulators are 
approximate replicas of some real world system, they can be created with 
dozens or hundreds of different representations of the world which may or 
may not be accurate and useful models of the real system and the purpose 
to which they are being put. This process seeks to expose the degree to 
which the simulator hardware, software, and data effectively represent the 
real world. This has to be done in the context of the expected application 
of the simulator. This context is essential in deciding whether 
compromises which have been made impact or invalidate the usefulness 


of the simulator in its specific application. icaaiases 
Figure 3. VV&A in Simulator 
Sargent’s framework has become the de facto validation process in the Development (Sargent, 2000) 


engineering and development of simulators. It is included in multiple 

later works which prescribe the process of simulator development and the accompanying validation of the product, 
such as Tolk (2012), Fishwick (2007), and others. In spite of this prevalence, the Sargent framework does not appear 
as a reference or an application in any of the medical simulation literature. Those communities come to simulation at 
a very different time in the system’s lifecycle. They more typically encounter a simulator after it has been designed 
and manufactured for them by a device company. The users of the simulator are then more interested in the degree 
to which it can assist them with teaching concepts and measuring competence, So their need for validation is entirely 
at the user experience, educational effectiveness, and student assessment levels. In spite of the fact that the device 
company may have rigorously applied VV&A ala Sargent (2000) and Tolk (2012), the medical users will insist upon 
another layer of validation of the product using one of the other frameworks. 


Classical Validation 


To support the needs of communities using educational devices, to include simulators, the American Educational 
Research Association (AERA) and the American Psychological Association (APA) proposed a framework for 
assessing educational tools, typically referred to as the “classical” framework (AERA, 1985). The goal of this 
validity model is to assess educational tools to ensure that a tool is meeting the educational goals of assessing the 
specific abilities that it was intended to test. 


Under this methodology. evidence is gathered to support a specific inference being made from test scores. For 
example, if a passing test score implies that a surgeon has the basic skills required to perform the removal of a 
prostate, then evidence would need to be gathered to support this claim. Under this framework, evidence is grouped 
into three categories: content related, criterion related, and construct related (Table 1). 
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Table 1. Summary of the Classical Framework 


Construct ‘A measure indicating the degree to | What is this test supposed to measure? 
which a test assesses the construct 
that it is intended on measuring. What is this test actually measuring? 
Content ‘A measure of the degree to which a | What is the content that needs to be tested? 
test's content represents a defined 
universe or content domain. Is the test content representative of the actual content? 


Does the response type and testing format match the 
universe? 


Criterion ‘A measure of the degree to which | Can the test scores accurately 
the test scores are related to one or | performance in the real world? 
‘more outcome criteria 


predict future 


How accurately can the test predict criterion 
performance? 


For construct related evidence, information is gathered to support that the test evaluates the specific characteristics 
of the quality being measured (i.e. does the test evaluate what it is designed to). The construct of interest is often 
ingrained in the test’s conceptual framework and is specific to the construct’s meaning, distinguishing it from other 
constructs and indicating how the measure should relate to other relevant variables. Gathering evidence in this 
domain may also involve evaluating aspects such as test format or administration, if these circumstances affect the 
test meaning and interpretation. 


Content evidence should demonstrate the degree to which test items, tasks, or questions are representative of a 
specified universe or area of content, given a proposed use of the test. Gathering evidence in this domain implies 
determining the content that needs to be tested and determining if the test is representative of that specific content. 
This also includes evaluating if the testing format and response mechanism is appropriate for the content (e.g. How 
is a student being assessed for a test on manual skill as opposed to a critical thinking). This type of evidence often 
relies on expert judgment to assess the relationship between the test and the defined universe, however observation 
in combination with expert input is acceptable. If a test is going to be used in a way that was not originally intended, 
the appropriateness of original domain definition needs to be evaluated for the new use. 


Criterion evidence demonstrates that test scores are systematically related to one or more relevant outcome criteria, 
The relationship between test scores and criterion measures may be expressed in several ways, with the goal of 
determining the accuracy to which the outcome criterion performance can be predicted from scores on the test. In 
general, there are two designs for obtaining criterion related evidence: concurrent and predictive methods. A 
predictive study obtains information supporting the accuracy with which test data can be used to estimate future 
criterion performance. A concurrent study serves the same purpose, but it obtains prediction and criterion 
information simultaneously. 


MeDougall (2007) adapted this framework for applicability to medical simulators. Under this modified framework 
the validation types included face, content, construct, concurrent, and predictive validity. Face validity is typically 
assessed informally by users and indicates whether the simulator is an accurate representation of the actual system 
(ie. the realism of the simulator). Content validity is the measure of the appropriateness of the system as a teaching 
modality. Experts who are knowledgeable about the device typically assess this via a formal evaluation. Construct 
validity is the ability of a simulator to measure what it is intended to measure. Often this is characterized by the 
simulator’s ability to differentiate between users’ experience level. Concurrent validity is the extent to which the 
simulator correlates with the “gold standard” for training and predictive validity is the extent to which the simulator 
can predict a user's future surgical performance. Collectively, concurrent and predictive validity are known as 
criterion validity and are used as measures of the simulator’s ability to correlate trainee performance with their real 
life performance. Face and content validity are most effective in evaluating the ability of a simulator to train a 
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surgeon; however construct, concurrent, and predictive validity are most useful for evaluating the effectiveness of a 
simulator to assess a trainee. The majority of literature surrounding the validity of surgical simulators uses these 
categories defined by McDougall. 


Unitary Validation 

The AERA and APA updated the classical framework to create a new methodology for validating educational tools, 
referred to as the “unitary” framework because it views validity as a unitary concept of five sources of evidence: 
content, response process, internal structure, relations to other variables, and consequences (Table 2). The more 
evidence collected, the stronger the validity argument is for the test for a specific interpretation, at any given time, 
for a specific population. Similar to the classic framework proposed by the AERA and APA in 1997, the assessment 
itself is not considered completely valid or invalid, but is more or less valid. 


Table 2. Summary of the Unitary Validation 


Test Content ‘A measure of the degree to which the test's | Are the test items assessing the content and 
content aligns with the content domain and | skills that they should? 
interpretation of scores. 
‘Response ‘A measure of the degree to which the response [Are test takers demonstrating the skills 
Process mechanisms of the test represent the skills | being assessed? 
being tested. 
Tnternal ‘A measure of the degree to which the format | Ts the test organized as it should be 
Structure and interrelatedness of the test items aligns 
with the construct being measured. 
Relation to ‘A measure of the degree to which the scores [Do the scores align with a test that is 
Other Variables _| are related to variables outside of the test. currently the gold standard? 
Consequences | A measure of the potential consequences of | Are the consequences of the test scores 
administering the test. relevant to the test's validity? 


Test content evidence refers literally to the content of the test being administered, For the purpose of this measure, 
“content” refers to the test items, to include the wording and formatting of the test, and procedures for 
administration and scoring. The evidence in this domain includes either a logical or empirical analysis of the 
adequacy to which the test content represents the content domain and of the relevance of the content domain to the 
proposed interpretation of test scores. For task-based assessments, as in the case of many simulators, test evaluators 
create a list of tasks required by the job via observation and advisement of a subject matter expert (SME). The SME 
judgment assesses the criticality and frequency related to the task performance. 


Response process evidence is gathered using a theoretical or empirical analysis of the response processes of test 
takers, which provides evidence in respect to the appropriateness of the construct and the nature of response 
mechanism used by the test takers, For example, if a test assesses critical analysis and reasoning, it is important to 
determine whether examinees are using this skill for the given material. The evidence for this domain is typically 
generated from an analysis of individual responses, including feedback from test takers regarding their performance 
strategies or reasoning of responses. In the case of scores being generated by evaluators, evidence can be gathered 
from the evaluators by determining the extent to which the evaluators are consistent with the interpretation of scores. 


Internal structure evidence indicates the degree to which the relationships among the test items comply with the 
interpretation of the test score. Evidence gathered for this domain would indicate if the items on the test support the 
assumptions of the interrelatedness of the items. For example if all items on a test will form a comprehensive score, 
then the test items should be one-dimensional. Test items may imply several aspects of a construct being tested and 
evidence in this domain determines the extent to which the items’ relationships align with the nece: 
framework 


Evidence gathered in regards to the relationship to other variables assesses the relationship of the test score to 
variables that are external to the test. The external variables can include measures of criteria that the test is expected 
to predict and relationships to other test scores that are expected to be either convergent or discriminant (i.e. 
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measuring the same or different constructs respectively). This evidence addresses questions about the degree to 
which these relationships are consistent with the construct underlying the proposed test interpretation. 


Lastly, evidence regarding the consequences does not necessarily affect the test's validity, but helps to inform the 
process of assessing validity. Evidence in this domain determines if there is a consequence of administering the test 
and if this consequence is relevant to other domains of validity. A finding in this domain of validity is relevant to the 
validity of the test in general if it can be directly related to another source of validity. 


SYMBIOTIC FRAMEWORKS 


When applying these frameworks to a simulation system being used for education, we can see that there is not one 
individually that meets all requirements of a system. While assessment is an essential component of a learning 
experience, it is not the only aspect that a user relies on for feedback when using a simulation system. Simulators are 
complex devices that often rely on the replicated 
controls and interfaces with real-world systems, 
including user feedback mechanisms (e.g. haptic 
feedback or visual stimuli). These mechanisms enhance 
user experience and facilitate learning by providing 
formative feedback and developing user expectations on 
how the real-world system should perform. Some 
simulators, including robotic surgery simulators, provide (cage) = 
summative feedback mechanisms to the user at the end So 
of the simulation experience, which helps to reduce the |" II } 

need for a proctor during the trainings (Figure 4). This or cr 
feedback is often given based on specific criteria and = 
benchmarks that are relevant to the task that the user is J | a 


performing, 


During the simulation experience, the user makes an 
input into the system and receives a corresponding Figure 4. Robotic Surgery simulator summative 
output from the system, For example, by moving a feedback screen 

camera control towards a target area, the field of view 

will change to the specified location. By receiving that output the user decides what the next input will be. Using the 
camera example, if the user over compensates and moves the camera past the target location, they would see this and 
use the camera control to adjust the field of view. This cycle continues until the simulation experience is complete 
(Figure 5). 


* The process of leaming via simulation is an 
— experiential process that can be related to the 
5 Kolb Experiential Cycle (1984) as shown in 
Simulator Assessment | Figure 6. When looking at this model, the 
ne im simulator plays a crucial role in the learning 
23} Mates experience of the user. The user expectations 

8 i ===. || are established during the concrete experience 
with the simulator. The learner applies that 


experience for reflective observation and to 
form an abstract conceptualization of how to 
improve performance. Thus, the user's learning is facilitated through their interactions with the system and the 
formative feedback that they receive from system. 


Figure 5. User interaction with simulator 


2016 Paper No. 16064 Page 8 of 11 


Interservice/Industry Training, Simulation, and Education Conference (W/ITSEC) 2016 


J system Vaidaton 


User Experence | 
{McDougal vaiaton | 


al 


Assessment 
and Feedback 
Roectve 
Expermentabon Observation 


Develop Strategy 
for Improvement 
Abstract 


Concerta 


Figure 6. Image showing the relationship of the three frameworks 


When looking specifically at the two educational models, the frameworks are designed for evaluating assessments 
and as such are focused on whether the assessment of the student was an accurate measure of the knowledge and 
skills that are being evaluated. If we only look at the assessment component of a simulator, then we are only looking 
at a small portion of the learning experience as a whole, It is possible to have a simulator that meets a high level of 
educational validity, but is not realistic in terms of engineering design. Conversely, we can have a simulator that 
almost perfectly replicates the intended system, but does not have meaningful associated metrics. In either case, the 
user would develop an incorrect model of their knowledge and skills during the training and assessment that would 
not translate to the real world system, 


These frameworks cannot individually address the comprehensive needs for validation of educational simulators and 
thus need to be used complementarily to one another. Table 3 provides an example of different degrees of validity 
according to each framework which can be used to evaluate the individual simulator components and to address the 
needs of educators comprehensively. 


Systems Engineering Output does not match Poor graphics Highly realistic 
Framework: the real world *  Pseudo-physics graphics 
measures. models. + Realistic physics 
models. 
Classical Framework + Replicates real-world | * Custom hardware that [* Embedded Simulator 
(McDougall) system to demonstrate is more realistic, but same hardware as in 
placement of controls, not exact. the real system, 
but do not function 
the same. 
Educational Framework [+ Test content doesnot | * The content aligns * Test content is 
align with content with the content relevant to the content 
domain. domain. domain. 
+ Test does not measure | * The users are not + Scores can predict 
what itis intended to. demonstrating the future performance 
necessary skills 
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‘CONCLUSION 


This paper summarizes three prominent and valuable frameworks and demonstrates the role that each takes in the 
validation process. These frameworks overlap to some degree, no one framework is a complete duplication or 
replacement of another. Thus, the goal is to explain the rationale for the decidedly different processes that are 
referred to by the same term and create an awareness of these methodologies, potentially provoking adoption or 
adaptation. Understanding the value of different frameworks may reduce arguments and contention between 
communities attempting to apply their own perspective to other communities. 


While valuable to specific fields, none of these validation models individually address the comprehensive needs 
when using simulation technologies as education and training tools. The learning experience when using a simulator 
encompasses components that should be evaluated distinctly to truly speak to the value of the system as an 
educational tool. Furthermore, disvaluing one aspect of the system during validation could have detrimental effects 
on the transfer of training for the user, potentially leading to negative training, 


The field of simulation integrates technology, processes, and ideas from several different communities, using 
technology-rich learning environments to provide learners with a real-world experience for practice and assessment. 
To say that one method of validation alone is sufficient would be naive. These frameworks were developed by their 
respective communities to address that community’s specific needs, however needs of the broader simulation 
community require a more interdisciplinary approach 


It is imperative to critically evaluate not only about what the validation is used for, but also what the validation is 
evaluating and leverage the qualities of each of the validation frameworks when assessing the validity of a system. 
We must consider the role that each framework plays in a system and how that affects the leamer. 


2016 Paper No. 16064 Page 10 of 11 


Interservice/Industry Training, Simulation, and Education Conference (VITSEC) 2016 


REFERENCES 
American Educational Research Association, American Psychological Association, National Council on 
Measurement in Education. (1985). Standards for educational and psychological testing. Amer Psychological Assn. 


American Educational Research Association, American Psychological Association, National Council on 
Measurement in Education, (1999). Standards for educational and psychological testing. Amer Educational 
Research Assn. 


Balci, O. (1997). Principles of Simulation Model Validation, Veri 
for Computer Simulation. 14, 3-12. 


tion, and Testing. Transactions of the Society 


Fishwick, P.A. editor. (2007). Handbook of Dynamic System Modeling. Boca Raton, FL: CRC Press. 


Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and development (Vol. 1). 
Englewood Cliffs, NJ: Prentice-Hall. 


MeDougall, E.M. (2007). Validation of Surgical Simulators. Journal of Endourology, 21(3), 244-7. 


Sargent, R.G. (2000). Verification, Validation, and Accreditation of Simulation Models. In Proceedings of the 
Winter Simulation Conference. Piscataway, NJ: IEEE Computer Press, 106-14, 


Smith, R.D., Truong, M., Perez, M. (2015). Comparative analys 
surgical robot. Surgical Endoscopy, 29(4), 972-83 


of the functionality of simulators of the da Vinei 


Stephanid 


, D. et al. (2015). Simulation in Surgery: What’s Needed Next? Annals of Surgery, 261(5), 846-53. 


Tolk A. editor. (2012). Engineering Principles of Combat Modeling and Distributed Simulation. New York: John 
Wiley & Sons. 


2016 Paper No. 16064 Page I of 11 


NICHOLSON CENTER 
The skill to heal. The spirit to care.® 


@ 


Abstract 


Background: Since the FDA approved robotically-assisted surgical devices for human surgery in 2000, the number of 
surgeries utilizing this innovative technology has risen. In 2014, approximately 570,000 robotic-assisted procedures 
were performed worldwide. Surgeons must be properly trained to safely transition to using such innovative 
technology. Multiple virtual reality robotic surgical simulators are now commercially available for educational and 
training purposes. There is a need for comparative evaluations of these simulators to aid users in selecting an 
appropriate device for their purposes. 


Methods: We conducted a comparison of the design and capabilities of all dedicated simulators of the da Vinci robot - 
the da Vinci Skills Simulator (dVSS) (Intuitive Surgical Inc., Sunnyvale, CA), dV-Trainer (dV-T)(Mimic Technologies Inc., 
Seattle, WA), Robotic Skills Simulators (RoSS)(Simulated Surgical Skills, LLC, Williamsville, NY), and the RobotiX Mentor 
(Simbionix USA Inc., Cleveland, OH). This paper provides the base specifications of the hardware and software, with an 
emphasis on the training capabilities of each system. 


Results: Each simulator contains a large number of training exercises, dVSS n=40, dV-T n=65, RoSS n=52, RobotixX 
Mentor n=31 for skills development. All four offer 3-D visual images but use different display technologies. The dVSS 
leverages the real robotic surgeon’s console to provide visualization, hand controls, and foot pedals. The dV-T RoSS, 
and RobotiX Mentor created simulated versions of all of these control systems. Each includes systems management 
services which allow instructors to collect, export, and analyze the scores of students using the simulators. 


Conclusions: This study provides comparative information of the four simulators functional capabilities. Each device 
offers unique advantages and capabilities for training robotic surgeons. Each has been the subject of validation 
experiments, which have been published in the literature. But those do not provide specific details on the capabilities 
of the simulators which are necessary for an understanding sufficient to select the one best suited for an organization’s 
needs. This review provides comparative information to assist with that type of selection. 


Background 


Since the FDA approved robotically-assisted surgical devices for human surgery in 2000, the number of surgeries utilizing this 


innovative technology has risen. In 2014, approximately 570,000 robotic-assisted procedures were performed worldwide [1]. 
Intuitive’s da Vinci surgical system is the currently the only FDA approved robotic-assisted surgical device (RASD) for human 
procedures. This system is an innovative surgical technology that allows 3-D vision, 7 degrees of freedom instruments, tremor 
damping, motion amplification, camera stability, and other features not currently available in traditional minimally-invasive techniques 
[2]. Investing in a robotic surgery program within a hospital can require significant resources and surgeons must be properly trained to 
safely adapt any innovative technology. A hospital considering this investment must account for equipment, facilities, personnel, and 
training. When evaluating the types of training systems to incorporate within the program, it can be difficult to make an informed 
decision about what system will work best for your hospital. Virtual reality (VR) simulation was first introduced to surgical education in 
the late 1980s. The first virtual reality robotic simulator was introduced in 2010 [3]. Multiple virtual reality robotic simulators are now 
commercially available for educational and training purposes. 


This is an extension of a previous analysis, which examined the functionality of three of the available simulators (i.e., da Vinci Skills 
Simulator, dV-Trainer, and Robotic Surgical Simulator) and illustrated the capabilities side-by-side for ease of evaluation by potential 
users of each device [4]. Since then, new technologies, exercises, and simulators have emerged [5]. The objective of this research is to 
provide comparative data on the functionality of the four commercially available robotic simulators: 

¢ da Vinci Skills Simulator (Intuitive Surgical Inc., Sunnyvale, CA); 

¢ dV-Trainer (Mimic Technologies, Inc., Seattle, WA) 

* Robotic Surgical Simulator (Simulated Surgical Skills LLC, Williamsville, NY):and 

* RobotiX Mentor (Simbionix USA Inc., Cleveland, OH). 


Methods 


The research division of Florida Hospital Nicholson Center purchased the four commercially available VR 
robotic simulators that are compared in this evaluation. We elected to purchase all of the systems to 
ensure that this evaluation remained objective and without undue influence from the manufacturers. 
The team reviewed the device manuals to collect details about each system [5-8]. Representatives of 
each of the manufacturing companies were also contacted for additional functional details. The authors 
also explored and compared the capabilities of each device by using each system at the Nicholson Center. 


Florida Hospital Nicholson Center, Celebration, FL 


Results 


FLORIDA HospitAL A Side-by-Side Comparison of Virtual Robotic Surgical Simulators 
Danielle Julian, Alyssa Tanaka, Ph.D., Patricia Mattingly, M.D., & Roger Smith, Ph.D. 


Each simulator utilizes a unique hardware and software for training and surgical rehearsal. All four offer 3-D visual images, but use different display technologies. The general features and capabilities are summarized in Table 1. The 

differences in hand controls across all the simulators is significant, so images of each of these systems are shown in Figure 1. All of the simulators allow an administrator or instructor to manage and organize student performance 

according to the unique login credentials of the student, which are provided via unique menu systems (Figure 2). Each simulator contains a large number of training exercises for skills development (dVSS n=40, dV-T n=65, RoSS n=52, 

RobotiX Mentor n=31). Tables 2-5 show the different exercise categories for the simulators. Each simulator provides on-system instructions for every exercise in the form of textual documents and narrated video demonstrations. All four 

simulators use the host computer to collect student performance data in multiple performance areas. Using this data, the simulator provides scores for various surgical skills and a overall composite score for the exercise. Examples of 

each of these scoreboards are shown in Figure 3. 
Table 2. dVSS Exercise Categories 


Table 3. dV-Trainer Exercise Categories 


Table 4. RoSS Exercise Categories 


Table 5. RobotiX Mentor Exercise Categories 


Surgeon Console Overview 


An introduction to the controls of the da Vinci robot. 


Endowrist Manipulation 1 


instruments. 


Basic hand movements and usage of the wristed 


Camera and Clutching 


Basic foot clutching for both the camera and the third arm. 


Endowrist Manipulation 2 


Intermediate use of the hands and wristed instruments. 


Energy and Dissection 


Use of the energy pedals and associated instruments. 


Needle Control 


surgical needle. 


Focused exercises for dexterous manipulation of a curved 


Surgeon Console 


An introduction to the controls of the da Vinci robot. 


Orientation Module 


Introduction to the surgeon controls of the da 
Vinci robot. 


Motor Skills 


Development of precise controls of the 
instruments, including spatial awareness. 


Overview 

Endowrist Basic and intermediate use of the hand controllers and wristed 

Manipulation instruments. 

Camera and Basic foot clutching for both the camera and the third arm. 

Clutching 

Energy and Use of the energy pedals and associated instruments. 

Dissection 

Needle Control Focused exercises for dexterous manipulation of a curved surgical 
needle. 

Needle Driving Repetitive exercises for needle driving. 


Troubleshooting 


Introduction to error recovery on the da Vinci robot. 


Basic Surgical Skills 


Instruction on handling a needle, using 
electrocautery pedals and instruments, and 
the use of scissors on the robot. 


Intermediate Surgical 
Skills 


Control of the fourth arm, blunt tissue 
dissection, and vessel dissection. 


Basic Robotic Skills and Tasks 


Fundamentals of Robotic Surgery 
Robotic Suturing 

Robotic Single-Site Suturing 
Stapler 


Robotic Essential Skills 


Instruction on handling a needle, 
Suturing and knot-tying, basic and 
intermediate use of the hand 
controllers and wristed instruments. 
Control of the fourth arm, blunt 
tissue dissection, and vessel 
dissection. 


Procedural Modules 


Specific surgical procedures in a 


Repetitive exercises for needle driving. Games Challenging and entertaining game environments to apply the skills Hysterectom 5 i 
Needle Driving . learned. ” u ily amen Stee ded), 
: = : = 5 = : 7 ; al : environment (guided or unguided). 
é Challenging and entertaining game environments to apply Suturing Skills Suturing exercises with needle, following suture, knot-tying, and tissue Hands-oniSuraical Vidco and Hapie guided letusion through Prostatectomy 
eS the skills learned. el res ‘at aie 
Suturing exercises with needle, following suture, knottying RTN VR exercises specifically build to match physical devices in use by the Training specific surgical procedures. Lobectomy 
Suturing Skills and tissue closure. Research Training Network of sites led by Lehigh Valley Hospital. 
Table 1. Robotic Simulator Feature Comparison Fi g le Si m | M G | | 
Features DVSS. dV-Trainer RoSS RobotiX Mentor y u re y u a to F a ste r O n t ro e rs 
System Manufacturer Intuitive Surgical Inc. Mimic Technologies Inc. Simulated Surgical Simbionix USA Inc. 
Systems LLC 
Specifications Depth 7” Depth 36” Depth 44” Depth 29” 
(Simulator only) Height 25” Height 26" Height 77” Height 55” 
Width 23” Width 44" Width 45" Width 37” 
120 or 240V power 120 or 240V power 120 or 240V power 120 or 240V power 
Specifications Depth 41” Depth 36” Depth 44” Surgeon Tower: 
(Complete System as Height 65" Height 59" Height 77” Depth 28" 
shown in Figure 1) Width 40” Width 54” Width 45" Height 56” 
120 or 240V power 120 or 240V power 120 or 240V power Width 30” 
Instructor Tower: 
Depth 27” 
Height 59” 
Width 28” 
120 or 240V power ci Ski . dV-Trainer . 
Visual Resolution VGA 1024 x 768 VGA 1024 x 768 VGA 640 x 480 HD 1280x720 da Vinci Skills Simulator RoSS RobotiX Mentor 


Components 


Support Equipment 


Exercises 
Optional Software 


Scoring Method 


Student Data 
Management 


Curriculum Customization 


Administrator Functions 


System Setup 
System Security 


Simulator Base Price 
Support Equipment Price 
Total Functional Price 


Customized computer 
attached to da Vinci 
surgical console 


da Vinci Si surgical 
console, custom data 
cable 


40 simulation exercises 
PC-based Simulation 
management 


Scaled 0-100% with 
passing thresholds in 
multiple skill areas 


Custom control 
application for external 
PC. Export via USB 
memory stick 

None 


Create student 
accounts on external 
PC. Import via USB 
stick 

None 

Student account ID and 
password 


$85,000 
$500,000 
$585,000 


Standard PC, visual 
system with hand 
controls, foot pedals. 


Adjustable table, touch 
screen monitor, 
keyboard, mouse, 
protective cover, custom 
shipping container 

65 simulation exercises 
Mshare curriculum 
sharing web site 


Proficiency-based point 
system with passing 
thresholds in multiple skill 
areas 

Export student data to 
delimited data file and 
graphical reports 


Select any combination 
of exercises. Set passing 
thresholds and conditions 
Create student accounts. 
Customize curriculum 


Calibrate controls 
PC password, 
Administrator password, 
Student account ID and 
password 
$99,200 
$9,800 
$109,000 


Single integrated custom 
simulation device 


USB adapter, keyboard, 
mouse 


52 simulation exercises. 
Video and Haptics-based 
Procedure Exercises 
(HoST) 


Point system with 
passing thresholds in 
multiple skill areas 


Export student data to 
delimited data file 


Select specifically 
grouped exercises. Set 
passing thresholds 
Create student accounts. 
Customize curriculum 


Calibrate controls 
PC password, 
Administrator password, 
Student account ID and 
password 
$126,000 
$0 
$126,000 


Surgeon Tower with 
visual system with hand 
controls, foot pedals. 
Instructor Tower to hold 
support equipment, 
standard PC, 

Monitor, keyboard, 
mouse 


53 simulation exercises. 
MentorLearn curriculum 
management system. 
Clinical procedures 
modules with optional 
team training. 
Proficiency-based point 
system with passing 
thresholds in multiple skill 
areas 

Export student data to 
delimited data file and 
graphical reports. Online 
access to trainee reports 
Select any combination 
of exercises. Set passing 
thresholds and conditions 
Create student accounts. 
Customize curriculum 


Adjust 3D viewer 

PC password, 
Administrator password, 
Student account ID and 
password 


$95,000 


Figure | 2. Comparative Simulator Exercise Menus 
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Figure 3. Comparative Scoreboards for Each Simulator 


RobotiX Mentor 


da Vinci Skills Simulator 


Conclusions 


Robotic surgery requires a unique set of surgical skills compared to conventional 
laparoscopic surgery. A surgical training device that can provide an automated, 
objective performance assessment is desirable and may be useful for proficiency- 
based training. Robotic surgery simulators can provide entry-level familiarization 
and skills development in a safe environment outside of the operating room. 

The aim of this research is to provide a comparative analysis of the four 
commercially available robotic simulators. The simulators described are complex 
systems, which are significantly less costly to acquire and operate than the actual da 
Vinci robotic surgical system. This analysis can be used to aid potential users, 
buyers, instructors, and trainers who have a need for a simulator. Prior publications 
have directly compared the RoSS, dVSS, and dV-Trainer. This is the first comparison 
to include the Robotix Mentor. 


dV-Trainer 


RoSS 


RobotiX Mentor 
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Simulator-based Multi-modal Task Decomposition of Robotic Surgical 
Technique for Vaginal Cuff Closure 
Patricia Mattingly, M.D,, Alyssa Tanaka, Ph.D., Danielle Julian, M.S, 
Anna Skinner, Ph.D., Roger Smith, Ph.D 


Study Objective: To identify the explicit cognitive and psychomotor steps 
necessary to complete a robotic vaginal cuff closure, particularly those difficult to 
master, using a task decomposition. The purpose was to enable exact step 
replication and outcomes for learners and create an expert model of performance to 
objectively compare novice performance. 


Design: Technique analysis. 
Setting: Surgical education and simulation center. 


Participants: Expert surgeons, expert simulation educators, expert cognitive 
psychologists. 


Interventions: Expert surgeons performed a vaginal cuff closure on the RobotiX 
Mentor, a virtual reality (VR) simulator of the da Vinci robot, allowing a repeatable 
testing environment. The steps were performed deliberately while psychologists 
recorded details of each action of the surgeon and robotic instruments, as well as 
deviations that may be considered alternative solutions or erroneous paths. The 
decisions required to progress the task forward were also captured. 


Measurements and Main Results: 
The results were constructed into flow charts demonstrating the sequence of 
actions and decisions surgeons must make. This analysis found three primary steps 
required to perform the task (Figure 1). The results identified a standard path of 33 
actions that experts follow to accomplish the cuff closure. The decomposition found 
eight actions as reasonable alternatives and 11 relevant decision points required. An 
example of the decomposition can be seen in Figure 2. 


Conclusions: To our knowledge, this is the first multi-modal task analysis for a 
robotic surgery task, as represented in a VR simulator. These results provide 
surgical educators with the explicit steps a trainee must master to accurately 
perform a cuff closure. This process may also be used to create task specific 
checklists for evaluation by an instructor. These results are the basis for computer 
algorithms, which will provide a task specific scoring system of the exercise and an 
intelligent, automated instruction for learners using the simulation. 
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Abstract In pursuit of improving the quality of residents’ 
education, the Southeastern Section of the American Uro- 
logical Association (SES AUA) hosts an annual robotic 
training course for its residents. The workshop involves 
performing a robotic live porcine nephrectomy as well as 
virtual reality robotic training modules. The aim of this 
study was to evaluate workload levels of urology residents 
when performing a live porcine nephrectomy and the virtual 
reality robotic surgery training modules employed during 
this workshop. Twenty-one residents from 14 SES AUA 
programs participated in 2015. On the first-day residents 
were taught with didactic lectures by faculty. On the second 
day, trainees were divided into two groups, Half were asked 
to perform training modules of the Mimic da Vinei-Trainer 
(MAVT, Mimic Technologies, Inc., Seattle, WA, USA) for 
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4h, while the other half performed nephrectomy proce- 
dures on a live porcine model using the da Vinci Si robot 
(Intuitive Surgical Inc., Sunnyvale, CA, USA). After the 
first 4h the groups changed places for another 4-h s« 
All trainees were asked to complete the NASA-TLX |-page 
questionnaire following both the MdVT simulation and live 
animal model sessions. A significant interface and TLX 
interaction was observed. The interface by TLX interaction 
was further analyzed to determine whether the scores of 
each of the six TLX scales varied across the two interfaces, 
‘The means of the TLX scores observed at the two interfaces 
were similar. The only significant difference was observed 
for frustration, which was significantly higher at the simu- 
lation than the animal model, 1 (20) = 4.12, p = 0.001. 
This could be due to trainees’ familiarity with live 
anatomical structures over skill set simulations which 
remain a real challenge to novice surgeons. Another reason 


sion. 


might be that the simulator provides performance metrics 
for specific performance traits as well as composite scores 
for entire exercises. Novice trainees experienced substantial 
mental workload while performing tasks on both the sim- 
ulator and the live animal model during the robotics course. 
The NASA-TLX profiles demonstrated that the live animal 
model and the MdVT were similar in difficulty, as indicated 
by their comparable workload profiles. 


Keywords Robotic surgery training - Mental workload 
NASA Task Load Index 

Introduction 

Robotic-assisted urologic surgery is predicted to continue 


to grow in usage in the coming years, and residents trained 
in urology will increasingly be expected to be proficient in 
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robotic surgery [1]. The complexity of robotic technology, 
its steep learning curve, and work-hour limitation of resi- 
dent trainees make incorporating robotic training into res- 
idency a challenging task. Experts suggest that learning as 
a bedside assistant for robotic surgery has a rapid plateau; 
many programs are now utilizing physician assistants and 
surgical technicians for bedside duties to free the residents 
for console training [2]. In high volume programs it 
remains difficult for residents to gain hands-on console 
time due to their insufficient skill set and the complexity of 
most procedures, 

Robotic simulation training tools can, therefore, be uti- 
lized by novice trainees to shorten the learning curve and 
improve operative skills in a low-risk environment. In 
pursuit of improving the quality of residents’ education, the 
Southeastern Section of the American Urological Associ- 
ation (SES AUA) hosts an annual robotic training course 
for its residents. This workshop involves training of basic 
laparoscopic surgery skills using virtual reality training 
modules of the Mimic da Vinci-Trainer (MdVT, Mimic 
Technologies, Inc., Seattle WA, USA) as well as training 
on performing a nephrectomy using a live porcine model. 
For simulation training to be successful, it is essential that 
it (1) practices the relevant skills and (2) matches the level 
of difficulty (workload demand) similar to the demands 
experienced in the “real” procedure. Thus, the goal for the 
present study was to assess whether the workload demands 
experienced in the virtual simulation training environment, 
which trains basic robotic surgery skills, match those 
experienced in when performing the live nephrectomy 
using a porcine model. 


Materials and methods 


Select residents from each of the 14 training programs of 
SES AUA were invited to Orlando, FL, for a 2-day robotics 
training course. Up to 3 residents were invited from each 
training program, and 21 participated in the training course. 
This cohort of residents represented a wider range of 
training and diversity in experience than in previous 
courses, being exposed to robotic surgery early at their 
home institutions. Volunteer faculty were recruited from 
SES AUA training programs. 

‘The 2015 annual SES AUA robotics training workshop, 
which is outlined in more detail below, involved training 
nephrectomy on a porcine model as well as training on the 
MAVT trainer. Participants’ workload we assessed at both 
interfaces (MdVT and live porcine model) using the NASA. 
Task Load Index (NASA-TLX). The NASA-TLX assesses 
workload along six dimensions: mental demand, physical 
demand, temporal demand, performance, effort, and frus- 
tration [3]. Each is measured on a 21-point sca 
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can range between 0 (“Very Low”) and 100 (“Very 
High”), see “Appendix 2”. 
‘The SES AUA robotics course is outlined below [4] 


Robotic course day 1 


A full didactic session was broken into three components, 
Component 1 covered the basics of robotic surgery 
including room set-up, bedside assistance, and console 
essentials. Component 2 covered several aspects of robotic 
kidney surgery including patient positioning, port place- 
ment, and surgical techniques. Component 3 focused on 
robotic prostate surgery including port placement and dif- 
ferent surgical techniques. Didactics were supplemented 
with surgical videos and discussions of difficult surgical 
scenarios and possible complications. 


Robotic course day 2 


The trainees were divided into two groups. Half were asked 
to perform. skill tasks on the Mimic da Vinci-Trainer 
(MdVT, Mimic Technologies, Inc., Seattle, WA, U.S.A) 
for 4 h using the dV trainer (version 2) while the other half 
performed set tasks in a live nephrectomy on porcine 
model using the da Vinci Si robot (Intuitive Surgical Inc., 
Sunnyvale, CA). After 4h the groups changed places for 
another 4-h session, 


Simulation section 


In the 4-h MdVT simulation session, trainees were first 
given a tutorial of the console and its functionality. The 
trainees then proceeded to complete five exercises with 
increasing difficulty and required skills. The first exercise, 
“pick and place”, involved simple movements of rings 
from one pole to another and is used to orient the trainee to 
the simulator. The second exercise, “peg board” is more 
advanced and required the trainee to clutch hand instru- 
ments while moving the camera, which involves coor 
nated hand and foot movements, The third exercise, “ring 
walk”, involved moving a ring over a curvy bar without 
touching the bar with any portion of the ring. This drill 
requires all the above skills as well as maintaining 
awareness and accuracy with the ring position at all time, 
The fourth exercise, “thread the rings”, involves passing a 
curved needle through rings positioned at different angles 
without touching the ring with any part of the needle. This 
drill teaches trainees good suturing technique. The last 
exercise, “tubes 2”, is the most challenging and realistic. 
This drill is designed to replicate performing an 
urethrovesical anastomosis. It utilizes all of the above skills 
including accuracy, coordination, and sufficient needle 
control. 
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Animal training section 


In the 4-h porcine model live surgery session, all trainees 
spent Ih performing cystostomies and cystorrhaphies. 
They then spent 30 min practicing port insertion and robot 
docking. Finally, for 2,5 h, trainees conducted a bilateral 
nephrectomy which included artery, vein, kidney and 
ureter dissections and ligation, 


Questionnaire 


Alll trainees were asked to complete the NASA-TLX 
L-page questionnaire following both the MdVT simulation 
and live animal model sessions. 


Results 


‘Twenty-one residents from 14 programs in the SES AUA 
participated in this course. Seventeen (80.9 %) had used a 
console during an actual surgical case, while four did not, 
The distribution of the different levels of training among 
the residents is shown in Fig. 1. Unlike previous years” 
courses when only senior or chief residents participated, 
this course included more junior residents. This reflects a 
shift toward early exposure to robotic surgery during 
urology training in most academic programs. The number 
of robotic surgeries performed or assisted by residents at 
different levels of training is shown in Fig. 2. Trainees’ 
satisfaction with their program robotic surgery training was 
assessed (Fig. 3). Of the 17 residents who performed actual 
robotic surgery, 7 (41.2 %) stated that the simulator 
replicates real-life robotic surgery, while 10 (58.8 %) sta- 
ted that it did not. 

The NASA-TLX scores were converted to a 0-100 scale 
with S-point increments. The raw TLX method was 
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Fig. 1 Robotic Simulator Questionnaire: question 1 results 


Frequency of Console Exposure During Actual Surgery by Urology 
Residents’ Level of Training 


= ae 


Pau PGA 
Level of Training 


Number of Exposures =0 #25 #2650 #51100 =>100 


Fig. 2 Robotic Simulator Questionnaire: question 4 results 
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Fig. 3 Robotic Simulator Questionnaire: question 5 results 


employed to eliminate the weighting process of the dif- 
ferent TLX scales. To assess the NASA-TLX data at two 
interfaces (simulator vs. animal model) for the different 
levels of training (year of residency), a 4 (training 
level) x 2 (interface) x 6 (TLX scales) mixed ANOVA 
was computed. The Greenhouse-Geisser correction was 
used to correct for violations of the sphericity assumption, 
‘The ANOVA indicated a significant main effect for TLX 
scales, F (3.91, 66.44) = 4.93, p = 0.002, Npwaiat = 0.225, 
as well as a significant interface by TLX scales interaction, 
F (3.73, 63.42) = 3.73, p = 0.016, Npusiaa = 0.166). None 
of the other main effects and interactions were significant, 
To further analyze the TLX main effects, Bonferroni-cor- 
rected repeated-measures 1 tests were computed to deter- 
mine which TLX scales differed significantly from each 
other; type-I error rate per comparison was set to 0,003. 
Means of the TLX scales are presented in Fig. 4. As can be 
seen from Fig. 4, effort resulted in the highest score. The 
Bonferroni-corrected tests indicated that mental demand 
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was significantly higher than physical demand 
[t (20) = 4.05, p=0.001] and then frustration 
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NASA TLX Seales 


Fig. 4 Mean scores of the NASA-TLX seales. Error bars refer to 
standard error of the mean 


Mean Score 
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Fig. § Mean scores of the NASA-TLX scales in simulation versus, 
animal model 


[+ (20) = 3.52, p = 0.002]. Further, temporal demand was 
significantly higher than physical demand [1 (20) = 2.90, 
p = 0,009] and effort was significantly higher than physi- 
cal demand [1 (20) = 6.52, p < 0.001], temporal demand 
[t (20) = 5.12, p < 0.001], performance [1 (20) = 5.15, 
Pp <0.001], and frustration [7 (20) = 6.90, p < 0.001] 

‘The analysis of the interface by TLX interaction was 
further analyzed to determine whether the scores of each of 
the six TLX scales varied across the two interfaces. On that 
end, Bonferroni-corrected repeated-measures 1 tests were 
computed; type-I error rate per comparison was set at 
a = 0.008, The means of the TLX scores observed at the 
two interfaces are in Fig. 5. The only significance was 
observed for frustration, which was significantly higher at 
the simulation than the animal model, t (20) = 4.12, 
p= 0.001. 
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Discussion 


Robotic surgery is increasing in popularity in the field of 
urology due to its minimal invasiveness, reduced risk of 
complications, and shortened hospital stay. This growing 
trend is evident in our results. The majority of the trainees 
this year (80 %) reported live console exposure. In contrast 
with a similar survey conducted in 2013 in a group of SES 
AUA trainees, only 56.9% of the trainees that year 
reported having had robotic console time [4]. During the 
2014 annual training course 92 % of the trainees reported 
performing live robotic surgery at their home institution 
[5]. Despite these increasing numbers, there is a lack of 
standardization and certification process for urology resi- 
dents in robotic surgery. Furthermore, there is no stan- 
dardized training protocol for residents learning robotic 
surgery across the various training programs. Gover et al. 
suggested a threshold of 25-30 cases for a novice surgeon 
to begin to operate the foot pedals and controls safely and 
intuitively [6]. Only 4 (19 %) of our trainees reported 
having performed more than 25 cases. 

Robotic surgery simulators have been proposed to na 
row the gap of novice trainees’ skill levels [7]. Such sim- 
ulators can help establish the basics of important operative 
skills such as eye-handfoot coordination and using the 
console controls and foot pedals. Our program chose to use 
the MdVT simulator for training. The Mimic da Vin 
‘Trainer (MdVT, Mimic Technologies, Inc., Seattle, WA, 
and USA) is one of the most established virtual robotic 
surgical simulators today. Previous research indicated that 
training on the MdVT resulted in superior surgical per- 
formance compared to solely training on the real da Vinci 
surgical system (Intuitive Surgical Inc., Sunnyvale, CA) 
when taking a robotic skills assessment using the real da 
Vinci system [9] 

‘The goal for the present study was to determine whether 
performance of the robotic surgery simulator tasks employed 
by the training course of the SES AUA matches the workload 
demands when performing a real robotic surgery. Towards 
that end, a porcine nephrectomy was employed. Thus, the 
results of the present study indicate that the simulation 
exercises employed by SES AUA generally induce similar 
workload demands to those experienced when performing a 
live porcine nephrectomy, indicating that the simulation 
exercises are not too easy. Specifically, the results indicated 
that mental demand and effort were major contributors of 
workload across both surgical interfaces, Further, the 
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different workload dimensions did not significantly differ 
across the two surgical interfaces, with the exception of 
frustration, Significantly higher frustration levels were 
observed at the simulation than the animal. Higher frustra- 
tion might be due to trainees being more familiar with alive 
anatomical structures than the simulation exercises. Another 
potential reason for the simulation to induce higher frustra- 
tion levels than the animal is that the simulator provides 
metrics for specific performance traits, as well as a composite 
performance score [7]. In addition to the objective metrics, 
the MdVT simulator defines thresholds which indicate 
whether the trainee’s score is considered a “passing” or 
“failing” performance with acceptable and warning scoring 
levels, respectively [7]. Conversely, the animal hands-on 
part did not have objective metric parameters to assess the 
skill set of trainees in robotic surgery. The faculty of the 
course subjectively evaluated the proficiency levels of resi- 
dents when they performed the porcine nephrectomy. Fur- 
thermore, the timeframe for every trainee was limited at the 
robotic console when performing the nephrectomy when 
compared to the simulation. 

However, though training on the MdVT simulator, has 
been validated [8], its use is not without limitations. There 
is an initial purchasing cost which ranges from $85,000 to 
$100,000. These are added costs of annual maintenance 
fees, There are currently no urology specific procedure 
modules or simulation drills available but only general 
surgical skill tasks like the ones used during the SES AUA 
training course. This limitation could hinder a rapid 
learning plateau and might not translate to better operative 
skills without supplementing with real live surgery console 
time. Therefore, work on more realistic 3D case simula- 
tions to advance clinical decision-making and procedural 
knowledge is currently in progress. The animal lab used for 
the course in this analysis cost roughly $1,900/h for the 
animal models, pharmaceuticals, veterinary support, 
robotic equipment with instruments, PPE, and the specially 
equipped facility. Other sites have reported $500/h, but this 
only includes the cost of the animal model, not the entire 
package of services and equipment [10]. It also lacks 
realistic human anatomy and might provide a false sense of 
security which could lead to harming a patient [11]. Future 
work should be invested in developing urology-specific 
training modules such as radical prostatectomy and partial 
nephrectomy simulations. The existing application only 
hones skills used in general robotic surgery and is not 
necessarily reflective of skills needed to perform urologic 
robotic surgery. 


Educators and companies have yet to determine the best 
model to use for teaching robotic surgery. Many factors 
must be taken into consideration including the cost, 
availability of expert faculty, legal responsibility on such 
supervising faculty, risk to patients, and the additional 
workload on trainees. 

These results of the present study, combined with pre- 
vious and future SES AUA training course results, can 
significantly enhance our efforts to establish a standardized 
robotic surgery training program that is cost-effective, 
practical, and of the highest quality. Encouraging the 
development of urology-specific robotic training tools in 
simulation will also aid in reaching our goal. Some limi- 
tations of this analysis include its regional focus and lim- 
ited sample size. It surveyed a limited number of trainees 
from the SES AUA and is not representative of trainees 
across the country. The analysis also did not assess the 
methods each program uses for robotic training. Upon 
completion of the residency program, many urologists 
recognize the effort and learing curve involved in acquiring 
robotic surgery skills and arrive at a consensus that training 
and proficiency in robotic surgery are necessary during 
residency [9]. Future direction for this project includes 
compiling detailed accounts of trainees’ exposures at their 
home institutes. Such analysis combined with future per- 
formance scores and trainees’ subjective opinions could 
lead to identifying the most effective methods of training. 
Work is currently in progress to improve the current 
robotic training methods. 


Conclusions 


Trainees experienced similar levels of workload when 
performing the virtual reality training modules and when 
performing a live porcine nephrectomy, indicating that the 
MAVT virtual reality training modules employed by SES 
AUA workshop have adequate difficulty. 
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Appendix 1: Robotic Simulator Questionnaire 
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What year urology resident are you? 

OD Uro-l 

O Ur2 

O Uro-3 

O Uro-4 

OD Uro-5S 

Does your training program own or have access to a robotics simulator? 
O No 

CO Mimic Simulator 

CO Ross Simulator 

1 Mimic Backpack or console 

O Other. 

Have you been on the robotics console for an actual case? 

O Yes 

O No 

Approximate the number of cases on which you have roboties console time 
oO <25 

O 26-50 

O 51-100 

 >100 

How do you rate your robotic training during residency? 

0 Poor 

O Fair 

1 Average 

O Excellent 

In your experience, do you feel that the simulator replicates real life roboties? 
O Yes 

O No 

Which drill did you find the most difficult? 

OO Peg board 

OO Ring Walk 

OO Thread the rings 

O Tubes 2 

If your program lacks a robotics simulator, do you think this device would be helpful in 
your program? 

O Yes 

O No 
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362 Appendix 2: NASA TASK Load Index 


Name Task Date 
Mental Demand How mentally demanding was the task? 
boriiiiii lipids 
Very Low Very High 
Physical Demand How physically demanding was the task? 
bertiiiittipiiiiyiity 
Very Low Very High 


Temporal Demand How hurried or rushed was the pace of the task? 
Very Low Very High 


Performance How successful were you in accomplishing what 
you were asked to do? 


EE eT ts a 


Perfect Failure 


Effort How hard did you have to work to accomplish 
your level of performance? 


Liter iiie titers ty 


Very Low Very High 


Frustration How insecure, discouraged, irritated, stressed, 
and annoyed wereyou? 


Ch fet ppp pe el ey eet py 


Very Low Very High 
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¢ The number of surgeries utilizing the da Vinci has 
risen with approximately 650,000 robotic- assisted 
procedures completed worldwide in 2015. 

¢ Investing in a robotic surgery program can require 
significant resources. 


¢ Evaluating types of training systems to incorporate 
alco elgeye]e-Waamer-lamel-meliui(ee|im 

¢ Multiple virtual reality robotic simulators are now 
FeWrel|r=Ve](omm ce) mm =1e[¥ler=Ui(e)areler-lale mire Vali ale olU|s ele \syoto 

Pm ©) o) (=XeudhY{= Hm ma ceyvale(omecelan) eF-le-li\v(omeralt- mela mlars 
capabilities and functionality of the four commercially 
available robotic simulators. 
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simuetors Compared _| ah 
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da Vinci Skills dvV-Trainer Robotic Skills RobotiX Mentor 
Simulator Simulator 


<e 
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Base Price: $85,000 Base Price: $99,200 Base Price: $126,000 Base Price: $95,000 
Support Equipment: $500,000 Support Equipment: $9,800 Support Equipment: $0 Support Equipment: $0 
Total: $585,000 Total: $109,000 Total: $126,000 Total: $95,000 
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System Administration 


¢ Simulators contain system configuration and student 
management functions. 

¢ Record and export student scores for evaluation and 
analysis. 

¢ dVSS administrator functions are fixed within the 
system and cannot be changed. 

¢ dV-Trainer, RoSS, & RobotiX Mentor: create new users, 
curricula, and set passing thresholds. 


¢ Robotic surgery simulators can provide entry-level 
familiarization and skills development in a safe 
environment outside of the operating room. 

¢ The simulators described in this study are complex 
training systems that are less costly to than the actual 
da Vinci system. 


FLORIDA HOSPITAL 
NICHOLSON CENTER 


¢ This is the first analysis to directly compare all 
four simulators. 

¢ This analysis can be used to aid potential 
users, buyers, instructors, and trainers who 
have a need for a simulator. 

¢ Which system Is right for you? 
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Response to “Unlike History, Should a Simulator Not Repeat Itself?” Simulation in Healthcare 2015; 
10(6): 331-335 
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407-303-4353 


Response to “Unlike History, Should a Simulator Not Repeat Itself?” Simulation in Healthcare 2015; 
10(6): 331-335 


To the Editor, 


Ihave read with great interest Dr. Lampotang's editorial in the December 2015 issue of Simulation in 
Healthcare [1]. The author has provided an excellent overview of some of the benefits of and the 
difficulty in achieving repeatability in healthcare training simulations. The categories and examples 
included should become common references for our unique niche of the simulation community in the 
years to come. The topic of repeatability has also been actively investigated in interactive, networked, 
and parallel simulation systems, often associated with military training applications. That community 
has literally created hundreds of simulation systems to address various problems and found 
repeatability to be important in applications like analytic wargames which needs to be run hundreds of 
times with very controlled differences in actions, but without uncontrolled variations from internal 
algorithms or data transfer times. Distributed simulation events which link multiple simulators via 
computer networks also encounter undesired repeatability issues primarily from two causes, differences 
in message delivery times from one run to another and the internal logic used to sequence events 
received from external systems which all have the same logical simulation time. These simulation 
communities have developed algorithms which specifically control for these variations and software 
infrastructures which attempt to provide these capabilities as a service to any simulator that uses them. 
(2,3) 


The definition that the author provides for repeatability is concise and useful from the perspective of 
the human users of the simulation. "Repeatability is the measure of the similarity in the outputs of a 
simulator during repeated runs of a given scenario with identical inputs, interventions, and events at the 
exact same times.” This correctly identifies the fact that identical output relies upon identical input in all 
of its forms. A simulation system of any significant complexity is prone to a large number of uncontrolled 
input factors and internal operations which make repeatability extremely difficult to achieve. This 
response will present some of the most common of these. 


When a simulation is a closed, single computer system that is driven only by preloaded digital data files, 
achieving repeatability is relatively straight forward. These systems can be said to be deterministic and 
can be structured to provide perfect repeatability, just as a calculator provides perfectly repeatable 
answers to the same problem every time. But when a simulation is part of an interactive, real-time 
experience that includes input from external systems like human participants and other computer 
devices, repeatability is much more difficult to insure, and may be impossible. 


For complex systems like these, repeatability can be explored at multiple levels. Lampotang explicitly 
identifies the model, simulator, and simulated environment [1]. He also provides examples of the 
information delivery between two devices, but does not list it with the other three. He has given several 
excellent examples where the linking of multiple devices and the interfaces between them can create 
uncontrolled variation which leads to non-repeatable outcomes. These various sources might more 
clearly be identified as stemming from external systems such as the humans, computers, and devices 
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that are part of the simulated environment; information delivery which includes computer networks or 
physical delivery lines that carry data or physical triggers to a simulator; internal interpretation by logic 
within the simulator that is used for managing and scheduling events that are internally generated or 
received from external systems; and internal models that use algorithms which may or may not provide 
repeatable results. Describing these sources or variation has been attempted in previous publications, 
though without the explicit terminology provided here [2,3,4]. Examples of how each of these can 
impact repeatability are provided below. 


When seeking repeatability, it is necessary to understand the basis of the internal models or algorithms 
which perform the computations within the simulator. In many fields, the lack of perfect knowledge of 
the domain (e.g. the human body) has led to the use of stochastic and statistical models to represent 
the richness and diversity of the domain. These models usually rely on a random number generator 
(RNG) as a source of input data. As the author points out, various races respond differently to 
anesthesia, as do different sexes, and body masses. The details needed to model this deterministically 
are often unavailable or too complex to include in a simulator. In these cases, simplified tables of 


average responses and standard deviations around those averages are often used along with stochastic 
and statistical algorithms which create variability within these defined limits [5]. Together these create a 
simulator in which a 40 year-old, Caucasian male, 6’0”, with a BMI of 25 does not always respond exactly 
the same to a volume of anesthetic, but always responds within known ranges for a person of that type. 
Such variability may be desirable for realism and uniqueness of training events, but is undesirable when 
repeatability is a goal. 


RNGs come in many forms, some provided as software libraries and others cleverly contrived by 
software programmers. In all cases, these actually generate pseudo-random numbers with a 
demonstrable level of bias or skew. Avoiding all use of RNGs and algorithms that depend on them is one 
step toward creating a repeatable simulation at the model level. RNGs found in software libraries often 
make use of a "seed number" which kicks off a long sequence of random numbers throughout the 
execution of the simulation. In these cases, deliberately using the same seed number at the beginning of 
every simulation event will lead to the same sequence of pseudo-random numbers throughout the 
event. However, this apparent repeatability can be thwarted by human actions and by system behaviors 
during a run, as will be explained. 


Inputs from an external system can also result in different computational outcomes. These systems may 
be both human users and external, networked computer devices. In both cases, the events which are 
generated externally and become inputs to a simulation can contain varying content which will throw 
off the repeatability of the simulator. These variations are much more common and extreme when the 
external system is a human trainee or instructor, but also occur when they originate from another 
simulator or device. When the events contain different contents due to slightly different actions taken 
externally, this information can easily lead to different decisions in the receiving simulation, change its 
internal state variables, and pass that variation on to the human trainee in its output. For example, 
when an injection of adrenalin is required to stimulate the heart, if the injection is provided only one 
minute later from one trial to the next, the simulator may cross an internally programmed threshold in 
the software. When the adrenalin is applied before this threshold is reached the patient may live, when 
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applied afterward the patient may die, even when the difference in the two times is only a few seconds. 
Such an extreme boundary may not be typical of most real life situations, but it is very common for 
software algorithms and data tables which are programmed into all types of simulations, creating these 
types of hard thresholds. 


Variation can also be triggered when external events are received with exactly the same internal 
content, but arriving at a different time or in a different order. When this occurs the simulator may 
receive and process event B before event A (B < A), rather than A < Basin a previous run. This reversal 
of order can be caused for multiple reasons; most commonly because the events were actually 
generated at a different time by a human user, but also when a different computer system is not strictly 
synchronized and can send events at different times during a second or third run of the simulation 
scenario. Also, when two or more simulators are linked together electronically, if one simulator 
generates multiple events at the same simulated time, these events may be delivered to another 
simulator in the same order, but because they have the same time stamp on them, they can logically 
and correctly be processed in either sequence A < B or B< A, which contributes to non-repeatability. If a 
controlled RNG is being used as described above, even the use of the same seed number on multiple 
runs cannot prevent this reversal of event order from reversing the application of the RNGs which were 
used on a previous run. There are several advanced parallel and distributed simulation infrastructures 
which can be used to insure that multiple simulators are synchronized and events are always processed 
in the same order [3,6]. These include infrastructure software like SPEEDES which was developed by the 
NASA Jet Propulsion Laboratories and is available as a product from WarplV Technologies; and the High 
Level Architecture (HLA) Runtime Infrastructure (RTI) which was designed by the US Department of 
Defense and is available as a product from multiple companies (e.g. VT MAK Inc., Pitch Technologies 
Inc.) These can eliminate variation within computer simulators, but they cannot correct or control the 
variation caused by human input. 


Medical simulation often looks to the military as a front runner in simulation techniques and 
technologies. Flight simulators, which were cited by the author and are widely understood by the public, 
are actually some of the most rudimentary and straightforward of these systems. As the author points 
out, these simulate the behavior of machines which has been engineered by humans and are much 
better understood than the human body. But, there are many military simulators that include models 
human behavior (e.g. OneSAF and SOAR), acting as individuals and groups; all of which wrestle with 
complexities similar to that found in modeling human physiology. Attempts to represent these 
behaviors have led directly to the creation of new fields of study or have expanded on existing fields — 
such as stochastic modeling, agent-based modeling, artificial intelligence, and machine learning. Some 
of these techniques may be useful in modeling human physiology and its response to various external 
medical and trauma stimuli. 


In summary, when humans and other sources of external stimuli are part of a simulation driven event, 
there are almost always sources of variation which can and will lead to non-repeatable runs of the 
scenario. Controlling as much of the externally generated stimuli as possible is the best option for 
approaching repeatability. It can lead to scenarios which are indistinguishable from each other most of 
the time, even though their internal state variables may have many differences. But, on occasion, these 
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differences will cross important thresholds which will lead to very different outcomes. True repeatability 
requires a level of control of the internal models, internal interpretation, information delivery, and 
external systems which is very difficult and costly to achieve. Recognizing when unexpected variation 
has changed the outcome of either training or assessment using a simulator is the responsibility of 
experienced human proctors and trainers. Simulated environments remain an approximation of the real 
world, but they also contain a level of complexity which makes them as difficult to control as it is to 
control the real world. 
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For centuries, the cognitive ergonomics of all learning environments have been driven by the 
availability of resources for training and education, rather than aligned with the best methods of 
transferring knowledge and skills from an expert to a novice. This Industrial Age education has 
become ingrained in all major education systems and is the basis from which most curricula are 
built. Medical education has followed suit for centuries, using apprenticeships and experiential 
learning on actual patients for skill acquisition. In these technologically sparse educational 
environments, the repercussions from an error were considered unfortunate for the patient, but 
accepted as a good for society as a whole. 


However, 21st century technologies provide alternatives for training medical providers, which shift 
the risks from human patients to computer and media-based tools that are much more resilient to 
mistakes made during the learning process. These tools not only allow hands-on practice in a 
safe environment, but also provide the repeated practice necessary to refine skills. With the 
introduction of such technologies and related objective skill assessments, the field has also seen 
a push for proficiency-based evaluation of skill for practitioners. 


The impact of technology driven educational devices on education and training is just beginning. 
These tools appear to offer higher levels of learning and skill acquisition with fewer negative 
impacts on patient safety, practitioner’s learning, and accreditation. In this session we will 
describe the advantages to practitioners, patients, and accreditation authorities of making more 
aggressive use of computer-based simulators in the education process. 
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Virtual Reality Robotic Simulation Performance Assessment: Simulator Metrics vs. 
GEARS 


Study Objective: To compare the performance metrics of virtual reality (VR) robotic 
surgical simulators to the Global Evaluative Assessment of Robotic Skills (GEARS) 
metrics for basic robotic tasks. The goal was to determine if a difference exists between 
the scoring mechanisms of both validated tools for measuring surgical expertise. 


Design: Prospective randomized study 
Setting: Surgical education and training center 
Patients: Residents, fellows, and attending surgeons (n=18) 


Interventions: Participants were randomized to a specific order in which they used the 
dV-trainer and da Vinci Skills simulator (dVSS). The subjects performed two warm-up 
exercises: Pick & Place and Basic Camera Targeting and then completed two trials of 
Ring & Rail 1 (RR1) and Suture Sponge 1 ($S1) on each simulator. The simulator 
performance was video recorded and the de-identified videos were sent to expert robotic 
surgeons to review using GEARS. 


Measures and Main results: The subjects’ demographics can be seen in table 1. 
‘The second trial of RR1 and SS1 was used for video review and simulator evaluation. 
The ranges of individual simulator metrics were calculated and quintiles were 
identified. The simulator metrics were then categorized into the appropriate 
quintile, allowing comparison to the Likert GEARS scoring system. 


Ina preliminary analysis, the Economy of Motion simulator metric and the GEARS 
Efficiency metric were compared for both the dV-Trainer and dVSS. No differences 
were found between Efficiency and RR1 or SS1 in either system (Table 2). 


Conclusion: 
To our knowledge, no study has compared simulator metrics to video evaluations 
performed by experts. The preliminary data suggests that no difference exists 
between simulator scores and GEARS. Since the simulator’s evaluation is similar to 
an expert robotic surgeon for basic tasks, this may indicate that a human evaluator 
may not be necessary for assessment of these skills. The final analysis is pending 
and will be completed by AAGL 2016. 


Video Game Impact on Basic Robotic Surgical Skills 
Alyssa Tanaka, M.S, Courtney Graddy, MHA, Manuela Perez, M.D, Ph.D., Khara 
Simpson, M.D,, Mireille Truong M.D., Roger Smith, Ph.D 


Study Objective: To compare the performance of “expert” video gamers to medical 
students, “laypeople,” and expert robotic surgeons in a robotic surgery simulator. 


Design: Prospective study. 
Setting: Surgical training centers, medical schools, and videogame colleges. 


Participants: Video gamers (n=40), medical students (n=24), laypeople ((n=35) 
and robotic surgeons (n=6) were recruited. 


Interventions: Subjects completed a demographic questionnaire and three 
computer-based perceptual tests: a Flanker compatibility task, subsidizing task, and 
Multiple Object Tracking. Participants performed two warm-up exercises and eight 
trials of two exercises (i.e. Ring & Rail 1 and Basic Suture Sponge) on the Mimic dV- 
Trainer. Participants then completed the NASA Task Load Index and a post- 
questionnaire. 


Measurements and Main Results: Gamers improved significantly from trial one to 
trial eight for all metrics in Ring & Rail 1 (Overall Score (OS) p=0.000; Economy of 
Motion (EoM) p=0.000; and Time p=0.000) and Suture Sponge (OS p=0.000; EoM 
p=0.000; and Time p=0.000). 


Surgeons performed better on trial one of Ring & Rail for all metrics (OS p<0.05; 
EoM p<0.05; and Time p=0.002). Gamers performed significantly better than 
laypeople on trial eight of Ring & Rail for Economy of Motion (p<0.05) and Time 
(p<0.05). Surgeons performed significantly better for all metrics (OS p=0.000; EoM 
p<0.005; and Time p=0.00) in the first and eighth trials (OS p<0.05; EoM p=0.001; 
and Time p=0.000) of Suture Sponge. Medical students outperformed gamers for 
Time (p=0.05) in the first trial. No other differences were found. 


Conclusions: A preliminary analysis indicates that videogame experience may 
influence the acquisition of basic, but not complex robotic surgical skills. The results 
could have implications for surgical training and other fields, indicating that 
consideration should be paid to the application of skills developed through other 
technologies. 


This study is ongoing and a final analysis will be available for presentation. 
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Abstract—Virtual reality simulators have emerged as 
valuable tools for standardized and objective robotic surgery 
skill training and assessments. In recent years the idea of using 
video game technology in surgical education for laparoscopy has 
also been explored, however few have attempted to make a 
connection between video game experience and robotic surgical 
skills. Thus, the current study aims to examine the performance 
of video gamers in a virtual reality robotic surgery simulator. 
Furthermore, the video gamers’ performance was compared to 
that of medical students, expert robotic surgeons, and 
“laypeople.” The purpose of this study is to demonstrate that 
video gamers acquire perceptual and psychomotor skills through 
video game play, similar to those used by robotic surgeons. 


Subjects completed a demographic questionnaire and 
performed three computer-based perceptual tests: a Flanker 
compatibility task, a subsidizing task, and a Multiple Object 
Tracking test. Participants then performed two warm-up 
exercises and eight trials of two core exercises on a robotic 
surgery simulator. After completing all trials, participants 
completed a post-questionnaire regarding their experience with 
the system. Expert video gamers (n=40), medical students (n=24), 
laypeople (n=42) and expert robotic surgeons (n=16) were 
recruited. Medical students and gamers were significantly faster 
than experts in the Flanker Task. The experts were significantly 
slower than the all other groups in the subsidizing task. Experts 
scored significantly higher, were significantly more efficient, and 
were significantly faster than laypeople, medical students, and 
gamers in the first trial of Ring & Rail 1 and Suture Sponge. In 
trial eight of the simulation exercises, the experts performed 
significantly better than most groups in all of the metrics. 


Contrary to prior literature in laparoscopy, this study was 
unable to validate enhanced abilities of video gamers in a robotic 
surgery simulator. This study does further demonstrate that the 
transfer of skills developed through video game play is relevant 
to the surgical technique. This may be due to the differences of 
the systems and how the users interact within them. In a society 
where video games have become an integral past time, it is 
important to determine the role that video games play in the 
perceptual and psychomotor development of users. These 
findings can be generalized to domains outside of medicine that 
utilize robotic and computer-controlled systems, speaking to the 
scope of the gamers’ abilities and pointing to the capacity within 
these systems, 
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I, BACKGROUND 


Robotic surgery is an innovative approach to surgery, 
which introduces a new dimension to the surgical toolbox, 
Many systems preceded the current state of medical robotics, 
all of which were developed with a universal goal of 
facilitating the human surgeon in performing easier and more 
standardized surgeries (Figure 1). As variations of medical 
robots emerged, the Defense Advanced Research Projects 
Agency (DARPA) was concurrently developing the Green 
Telepresence System [1][2].This system evolved from the idea 
of a surgeon performing a procedure from a location remote to 
the patient, a concept referred to as telesurgery. The ability to 
perform surgery-at-a-distance has not completely come to 
fruition; however, it was realized that this concept is valuable 
not only for surgeons separated from their patients by 
hundreds of miles, but also a layer of skin, 


FIGURE T History of surgical robotics 


Currently the only FDA approved robotic system for 
a procedure on a human is Intuitive’s da Vinci Surgical 
System. This system consists of three main components: the 
surgeon console, patient cart, and video tower. The surgeon 
manipulates the master controllers at the surgeon console, 
which is communicated to the patient cart through a fiber optic 
cable connection and moves the robotic arms in an identical 
manner (Figure 2). While this system offers many advantages 
over other minimally invasive surgery (MIS) techniques, it 
also introduces a need for specialized training. Virtual reality 


(VR) simulators emerged as economical training tools that 
offer robotic surgeons standardized and objective skill training 
and assessments, In recent years, the role of video game 
technology in surgical education has also been explored. 


FIGURE 1. Data flaw from the da Vinel robot 


Previous research has demonstrated that trainees with 
prior video game experience perform better on basic 
laparoscopic tasks in a dry lab and VR taining environment 
[31[41[SI{6]. Studies have also investigated using video games 
as training tools for laparoscopic surgery, many of which 
found positive training outcomes [7I[8I[9][10][11]. Certain 
genres of video games have established effects on perceptual 
skills similar to those required by robotic surgeons, yet few 
have attempted to make a connection between video game 
experience and robotic surgical skills [12][13][14[15]. 


Thus, the current study aims to examine the 
performance of video gamers in a VR robotic surgery 
simulator. Furthermore, the video gamers’ performance was 
compared to that of medical students, expert robotic surgeons, 
and “laypeople.” The purpose of this study is to demonstrate 
that video gamers acquire perceptual and psychomotor skills 
through video game play, similar to those used by robotic 
surgeons. 


Il, METHODS 


A. Recruitment 


Participants in this study included video gamers, 
expert robotic surgeons, medical students, and “laypeople” 
(ce, individuals without formal medical education or extensive 
gaming experience), Gamers were recruited from a local 
university offering degrees specializing in game design and 
development (Florida Interactive and Entertainment Academy 
[FIEA]}). Expert robotic surgeons were recruited from Florida 
Hospital, Florida Hospital Nicholson Center training courses, 
and at relevant surgical conferences. These individuals were 
surgeons who self-reported performing at least 100 robotic 
surgical procedures, of which at least 50% of the procedure 
was spent on the surgical console, Medical students were 
recruited from the University of Central Florida College of 
Medicine (UCF CoM) and laypeople were recruited from all 
data collection sites, 


B. Design 


All subjects completed a pre-questionnaire and then 
performed three computer-based perceptual tests: a Flanker 
compatibility task, a subsidizing task, and a Multiple Object 
Tracking (MOT) test. The Flanker compatibility test required 
the participant to indicate the orientation of a single arrow in 
the center of a group of several other arrows, The arrows are 
randomly generated to all face the same orientation 
(congruent) or the opposite direction (incongruent) of the 
target arrow in the center. This tests attentional capacity by 
requiring the subject to focus solely on the relevant arrow and 
ignore other stimuli. The subsidizing task also assessed 
attentional capacity by requiring subjects to identify the 
number of dots that appear on the screen as fast as possible. In 
the MOT task, users tracked specific objects as they move 
across the screen with other identical objects, which assesses 
visual attention (Figure 3). 


FIGURE IL, The cognitive tests 


Participants then performed two warm-up exercises 
on a VR robotic surgery simulator, the Mimic dV-Trainer: 
Pick & Place and Basic Camera Targeting. This familiarized 
subjects with the system and system controls. All participants 
then performed eight trials of Ring & Rail 1 and Basic Suture 
Sponge to test various basic skills, which served as the 
primary sources for data analysis (Table 1). The Mimic dV- 
Trainer uses custom software and hardware to replicate the 
controls and viewing system of the actual surgical robot. This 
simulator was selected for this study from previously 
conducted research that confirmed the face, content, and 
construct validity of the system [16]. 


After completing all trials, participants completed a 
post-questionnaire regarding their experience with the system 
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(Figure 3). 


FIGURE IV. Study design 


‘TABLE I. Description of simulation exercises 


Exercise [Purpose | _ Objective | Still Trained 
Warm-up Exercises 
Pick & | Introduction to —] Place colored objects] —Endowrlst 
Place | usingstereo vision | inmatching colored | Manipulation 
‘and EndoWiist containers 
instruments for 
picking up and 
placing object. 
Basie | Learn to accurately} Manipulate the camara | Camera 
Camera | position the camera | 10 position light blue Sonal 
Targeting | while working ina | splire camera targets 
large workspace | in the center of your 
While practicing to | screen's dark blue 
keep the cxosshais, 
and developing 
stereo depth acuity 
‘Core Exercises 
Bing & | Coordinate control] — Pick up a ving and Tndowrlst 
Rail 1 ofan objec’s | guidethering along a | manipulation, 
position and curved rail Camera 
‘orientation along a Contrl 
trajectory using the 
EndoWrist 
Basie | Improve dexieniy | Insert and extacta | —Endowrist 
Suture | and accuracy when | needle through several | manipulation, 
Sponge | driving a needle | targets onthe edge of a | Camera 
through a sponge with random Control, 
deformable object. | variations in thelr Needle 
positions. Control, 
Needle Driving 


III, RESULTS 


Video gamers (n=40), medical students (n=24), 
laypeople (n=42), and expert surgeons (n=16) were analyzed 
in terms of demographic characteristics. The participants were 
primarily male (66%) and predominantly right-handed (88%). 
The average age of all subjects was 29 (SD=7.559). Fifty-five 
percent of all participants reported that they currently play 
video games. The gamers reported having played video games 
for an average of eighteen years (SD=5.71) and playing an 
average of twelve hours per week (SD=6.74). On average, 
expert surgeons performed 135 laparoscopic (SD=94.55) and 
95 robotic cases (SD=71.79) annually. Experts also reported 
performing an average of 1111 total laparoscopic cases 
(SD=725.41) and 624 total robotic cases (SD=607.13). Of the 


expert surgeons, 13% indicated that they currently play video 
games. Eighty-eight percent of expert surgeons reported that 
they have previously received formal robotic surgery training, 
with 94% of all expert surgeons indicating that have used a 
laparoscopic or robotic surgical simulator in the past. 


A. Perceptual Tasks 


Using a Kruskal-Wallis test, significant differences were 
found between the groups for the congruent time 
‘H(3)=18.297, p<0.001) and incongruent time (H(3)=14.865, 
p<0.005) metrics. No differences were found between the 
groups for the percent correct (H(3)=1.107). When looking at 
pairwise comparisons for the congruent time metric, the 
medical students were significantly faster than the experts 
(p<0.05) and laypeople (p<0.005). The gamers were also 
significantly faster than experts (p<0.05). Medical students 
(p<0.005), and gamers (p<0.05) were significantly faster than 
experts in the incongruent time metric. 


No significant differences were found between the groups 
for the percent correct metric in the subsidizing task 
H(3)=5.296, p=.151). Using ANOVA, significant differences 
were found between the groups for the time metric in the 
subsidizing task (F (3, 115)=4.711, p< .005). A Tukey post-hoc 
test revealed that the lay people (991.94 + 138,00 sec, p<.05) 
and experts (1058.59 + 120.87 sec, p<.005) took significantly 
more time than the gamers (921.40 4116.87 sec) to complete 
the task. The experts (1058.69 + 120.87 sec, p<.05) were also 
significantly slower than the medical students (957.99 + 148.45 
sec). Using a Kruskal-Wallis test, differences were found for 
the difficult level (p<0.05) of the MOT. No differences were 
found for the easy and normal levels (p=0.656 and p=0.130 
respectively). No significant pairwise differences were found 
for the groups (Table 2). 


B. Simulation Trial One 

Using a Kruskal-Wallis test, differences between the 
groups were found for the Overall Score (H(3)=12.90, 
p<0.01), Economy of Motion (H(3)=20.28, p<0.001), and 
‘Time to Complete (H(3)=32.55, p<0.001) metric of trial 1 of 
Ring & Rail 1. When looking at pairwise differences, experts 

scored significantly higher than laypeople (p<0.005), 
medical 


‘TABLE Il, Descriptives for the cognitive scores 


Flanker Subsidizing 
Percent Correct Congruent Time Incongruent Time Percent Correct Time 
9 100.00 42859 40432 81.39 921.40 
Comers (1QR=5.00) QR=72.66) (1QR=85.10) (1QR=15.99) (s0=116.87) 
Medical 100.00 414.92 460.28 7619 95799 
Students QR=2.50) QR=56.10) (QR=95.38) (1QR=22.65) (sb=148.45) 
100.00 439.91 509.73 76.19 991.94 
Laypeople (1QR=5.00) 4QR=85.47) (1QR=74.23) (1QR=16.66) (S0=138.00) 
100.00 Ring & Baily S60. ‘Sugure Sponge 1058.83 
s faR=5.00) (QR=98.80) (ar=91.25) (IQR=19.38) (Sd=120.87) 
Overall Economy of. Tithe 10 ‘Overall Economy of Motion Time t0 
7 core ‘Motion Complete “Ore Tomplet 
Category : ui iid 
40 69.08 67.02 439.26 517.88 
our gr=2574) | agr=35.33) | (sp=170.63) (0QR=307.11) 


‘TABLE IV. Descriptives for Trial 8 of the simulation exercises 


Ring & Raid Suture Sponge 
Overall Economy of Time w Overall Economy of Time 
a Score Motion Complete Score ‘Motion Complete 
Category 
@ease.s|"a0 118227 5061 2924 716.30 296.57 25072 
0QR=2.63) sp-1420) | sd=10.2) | aQR=537.76 | agR=10245) (s0-65.02) 
Medical | 2 114375 5204 2864 930.84 306.42 225.67 
Seulead (GQR=74.89) sp=16.71) | aQR=t099) | aQR=s80.61) ($0=76.12) (0-48.17) 
len 
Ly | 2 1106.08 6225 3337 695.93 30434 219.8 
f (0QR=7452) (spies | aQr=i7.665) | aQr=s07.58) agR-92.75) | aQR=109.02) 
people 
paige,  ci8 1161.96 36.38 127764 210.08 135.67 
Be aanesi7) | agaeiee7 agnizsy | sb-sir) | eoeanan) 
urgeon 


students (p<0.05), and gamers (p<0.05) in the Overall 
Score metric. Experts were significantly more efficient than 
medical students (p<0.005), gamers (p<0.001), and laypeople 
(p<0.001) in the Economy of Motion metric. Experts were 
significantly faster than laypeople (p<0.001), medical students 
(p<0.001), and gamers (p<0.001) in the Time to Complete 
metric. 

Significant differences were also found for the 
Overall Score (H(3)=28.31, p<0.001), Economy of Motion 
(H(3)=31.15, p<0.001), and Time to Complete (H(3)=39.62, 
p<0.001) metrics for the Suture Sponge exercise. When 
looking at pairwise differences, experts scored significantly 
higher than gamers (p<0.001), laypeople (p<0.001), and 
medical students (p<0.005) in the Overall Score metric. 
Experts were significantly more efficient than medical 
students (p<0.001), laypeople (p<0.001), and gamers 
(p<0.001) in the Economy of Motion metric. Experts were 
significantly faster than medical students (p<0.001), laypeople 
(p<0.001), and gamers (p<0,001) in the Time to Complete 
metric (Table 3). 


CC. Simulation Trial Eight 

Significant differences were also found among the groups 
for the Overall Score (H(3)=10.65, p<0.05), Econom3y of 
Motion (H(3)=20.99, p<0.001), and Time to Complete 
(H(3)=21.85, p<0.001) metrics for trial 8 of the Ring & Rail 1 
exercise. When looking at pairwise differences, experts scored 
significantly higher than laypeople (p<0.05) in the Overall 
Score metric. Experts were significantly more efficient than 
laypeople (p<0.001) in the Economy of Motion metric. Experts 
were significantly faster than gamers (p<0.05), medical 
students (p<0.05) and laypeople (p<0.001) in the Time to 
Complete metric. 


The groups also demonstrated significant difference for the 
Overall Score (H(3)=22.79, p<0.001), Economy of Motion 
(H(3)=23.62, p<0.001), and Time to Complete (H(3)=32.48, 


p<0.001) metrics for the Suture Sponge exercise. When 


looking at pairwise differences, experts scored significantly 
higher than laypeople (p<0.001) and gamers (p<0.005) in the 
Overall Score metric. Experts were significantly more efficient 


than medical students (p<0.005), gamers (p<0.001), and 
laypeople (p<0.001) in the Economy of Motion metric. Experts 
were significantly faster than medical students (p<0.001), 
laypeople (p<0.001), and gamers (p<0.001) in the Time to 
Complete metric (Table 4). 


D. Simulation Skill Acqui 


A closer evaluation of the groups’ ability to acquire basic 
robotic skills over the eight trials was conducted to evaluate if 
any one group improved significantly more than other groups. 
The difference between trial 1 and trial 8 was calculated for 
each participant and determined as the amount of change. A 
Kruskal-Wallis test was then used to determine if the amount 
of change was different between each of the groups. The 
groups demonstrated significantly different amounts of change 
for the Overall Score (H(3)=8.30, p<0.05) and Time 
(H(3)=25.84, p<0.001) metrics in the Ring & Rail 1 exercise. 
No differences were found between the groups for the 
Economy of Motion metric in Ring & Rail 1 (p=0.062). 


n 


‘When looking at pairwise comparisons, no differences were 
found for the Overall Score metric. Experts decreased their 
time significantly less than gamers (p<0.001), medical students 
(p<0.001), and laypeople (p<0.005) in the Time to Complete 
metric. Significant differences were also found between the 
groups for the Economy of Motion (H(3)=15.35, p<0.005) and 
Time (H(3)=24.78, p<0.001) metrics of Suture Sponge. No 
differences were found for the Overall Score metric. When 
looking at pairwise differences, experts improved their 
efficiency significantly less than gamers (p<0.005), medical 
students (p<0.05), and laypeople (p<0.05) for the Economy of 
Motion metric. Experts reduced their time significantly less 
than gamers (p<0.001), laypeople (p<0.001), and medical 
students (p<0.005) for the Time to Complete metric. 


TV. CONCLUSIONS 
Contrary to previous findings in laparoscopic 
research, the results of the current study were unable to 
confirm a relationship between playing video games and 
increased abilities in a robotic surgery simulator. The video 
gamers in this study did not perform better than laypeople or 
medical students in the perceptual tests. The expert surgeons 


outperformed all other groups in the first trial of the 
simulation exercises. They scored significantly higher, were 
significantly more efficient, and significantly faster than all 
other groups in the Ring & Rail 1 and Suture Sponge 
exercises. The gamers scored higher and were more efficient 
than medical students and lay people in Ring & Rail 1, but the 
differences were not significant. The gamers scored a lower 
score, were less efficient, and were slower than the medical 
students and laypeople in the Suture Sponge exercise. 


Experts scored significantly higher and were 
significantly more efficient than laypeople for the eighth trial 
of the Ring & Rail 1 exercise. The experts were also 
significantly faster than all other groups. Gamers and medical 
students surpassed laypeople in the Overall Score, Economy 
of Motion, and Time to Complete metrics, although the 
differences were not significant. In the Suture Sponge 
exercise, experts scored significantly higher than gamers and 
laypeople. The experts were also significantly more efficient 
and faster than all groups in this exercise. The expert surgeons 
improved significantly less than the other groups for the Time 
metric of Ring & Rail 1 and the Time and Economy of Motion 
metrics in the Suture Sponge exercise. It is likely that the 
experts have less room for improvement because they are 
already proficient in the tasks. 


While these results are conflicting with laparoscopic 
research, they align with the few studies that have examined 
the impact of video game play on robotic surgical skills 
[14][15]. So why does prior video game experience impact 
basic laparoscopic skills, but not robotic? Differences may be 
attributed to the distinctness of the systems in which the users 
are interacting. The skills developed in two-dimension video 
games may transfer more appropriately to laparoscopic 
surgery, which uses a two-dimensional screen, as opposed to 
the three-dimensional view in robotics. Laparoscopy also 
involves contrasting movements to the primarily fine motor 
movements of robotic surgery and it is possible that gamers 
are more inclined with the manual dexterity associated with 
laparoscopy. The movements associated with robotic surgery 
are also more intuitive than the proprioceptive challenge that 
laparoscopy presents. It is possible that gaming skills give 
users an advantage to overcoming the psychomotor 
difficulties, while the robot’s intuitive nature renders the 
advantage extraneous. 


In a society where video games have become an 
integral pastime, it is important to determine the role that 
video games play in the cognitive, perceptual, and 
psychomotor development of users. This research emphasizes 
the criticality in evaluating the impact of video games, prior to 
making assumptions on associated effects. Similar to the 
claims that video games are linked to violent actions, 
associations made without scholarly evidence can create 
inaccurate social associations [17]. This study has further 
emphasized that the effect of video game play on surgical 
skills is nuanced by the surgical technique. The findings can 
be generalized to domains outside of medicine utilizing 
robotic and computer-controlled systems, speaking to the 


scope of the gamers’ abilities and pointing to the capacity 
within these systems. The research regarding the use of video 
games for training in robotic surgery is nascent. 


Future research should further delve into the 
differences of the performance of gamers in laparoscopic and 
robotic tasks. Comparing gamers in both modalities would 
highlight the differences in performance. Research should also 
investigate if video game experience allows for better 
retention of skills, particularly leading to less skill degradation 
during periods of inactivity. 
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